robots.txt
Also called: robots.txt file, Robots Exclusion Protocol
Definition
robots.txt is a text file placed at the root of a site that gives instructions to crawlers: for each bot, identified by its name (user-agent), it lists the allowed and disallowed paths. Its format was standardized in 2022 by the IETF, as RFC 9309.
AI crawlers follow these instructions in principle, and each one has its own name: GPTBot and OAI-SearchBot for OpenAI, ClaudeBot and Claude-SearchBot for Anthropic, PerplexityBot, or Google-Extended for Google's AI uses. A single misplaced line can close your site to a whole engine.
Example
“User-agent: GPTBot” followed by “Disallow: /” blocks OpenAI's training crawler from the whole site. Many sites added these lines in 2023 as a precaution, without measuring what they cost in visibility today.
What it means for your AI visibility
Reread your robots.txt with the difference between training crawlers and search crawlers in mind. Blocking the latter means giving up on being cited in their engine's answers.
Free toolAI crawler simulatorSee your page the way GPTBot, ClaudeBot and PerplexityBot see it.Related terms
- AI crawlerAn AI crawler is a bot that browses the web on behalf of an AI company, to train its models or feed its answers.
- GPTBotGPTBot is the OpenAI crawler that collects pages to train its models. OpenAI uses other crawlers for ChatGPT's search.
- Google-ExtendedGoogle-Extended is a robots.txt token that lets you refuse the use of your pages for Gemini models. It has no effect on Google Search.
- llms.txtllms.txt is a file placed at the root of a site that introduces the brand to AI and lists the pages to read first.
Updated on · All glossary terms