See your page the way GPTBot sees it
Robots.txt, firewall and no JavaScript: what AI crawlers actually get.
10 AI CRAWLERS · ROBOTS.TXT, FIREWALL AND PAGE WITHOUT JAVASCRIPT
Three ways to disappear without knowing it
Your site works fine for a human. For an AI crawler, it can be closed, empty or hostile, and nothing warns you.
Your content arrives through JavaScript
Most AI crawlers don't run it. If your text is injected after the page loads, they leave with an empty shell.
Your robots.txt shuts the door on them
One Disallow left over from an old setting, or copied from a template, and a whole crawler is locked out. Nothing breaks for humans, so nobody notices.
Your firewall takes them for attackers
Some CDNs turn AI crawlers away by default. Your robots.txt says yes, your server answers 403.
Not all AI crawlers do the same job
Each AI company sends several crawlers. Blocking one doesn't have the same consequences as blocking another.
Training
GPTBotClaudeBotCCBotCollect pages to train future models.
If you block them: you stay in today's answers, but tomorrow's models will know less about you.
Search
OAI-SearchBotClaude-SearchBotPerplexityBotIndex your site for answers with sources, the ones that cite a page and link to it.
If you block them: you drop out of ChatGPT, Claude and Perplexity search.
On demand
ChatGPT-UserClaude-UserVisit when someone asks the AI to read a specific page.
If you block them: you turn away a visit a human just asked for.
Google-Extended and Applebot-Extended don't visit anything: they are robots.txt tokens that tell Google and Apple whether your pages can be used for their AI.
Is this really what GPTBot sees?
Almost. We read your robots.txt exactly the way it does, and we request your page with its identity, without running JavaScript, just like it does. The only difference: the request comes from our servers, not OpenAI's. A firewall that checks crawler IP addresses may answer the real GPTBot differently.
Should you let every AI crawler in?
Not necessarily, but know what you're blocking. Turning away training crawlers doesn't remove you from today's answers. Turning away search crawlers does. For a brand that wants to get cited, letting them all in is still the simplest option.
My page is built with JavaScript. Is that a problem?
If your main text only shows up after JavaScript runs, yes: for most AI crawlers, the page is empty. The fix is server-side rendering or pre-rendering, not a robots.txt setting.
Crawlers getting in? Give them a map.
Your llms.txt tells them what you do and which pages to read first. One address, one click, and the file is ready.
Generate my llms.txt