GPTBot and OAI-SearchBot
Also called: OpenAI crawler, GPTBot user agent
Definition
GPTBot is the crawler OpenAI has used since 2023 to collect pages that may be used to train its future models. It identifies itself with GPTBot in its user-agent and follows the instructions in robots.txt.
OpenAI uses two other crawlers. OAI-SearchBot indexes the web for ChatGPT Search: blocking it keeps a site out of ChatGPT's search results. ChatGPT-User comes by when a user asks ChatGPT to look at a specific page. So blocking GPTBot doesn't have the same effect as blocking OAI-SearchBot.
Example
A media outlet that doesn't want its articles used for training can block GPTBot while letting OAI-SearchBot through, so it keeps being cited in ChatGPT's answers.
What it means for your AI visibility
For a brand that wants to be recommended, blocking OAI-SearchBot is almost always a mistake. Blocking GPTBot is a choice, one that over time reduces what models know about you.
Free toolAI crawler simulatorSee your page the way GPTBot, ClaudeBot and PerplexityBot see it.Related terms
- AI crawlerAn AI crawler is a bot that browses the web on behalf of an AI company, to train its models or feed its answers.
- robots.txtrobots.txt is the file that tells crawlers, including AI crawlers, which parts of a site they're allowed to visit.
- ChatGPT SearchChatGPT Search is ChatGPT's web search feature: it reads pages when it answers and cites its sources.
- Google-ExtendedGoogle-Extended is a robots.txt token that lets you refuse the use of your pages for Gemini models. It has no effect on Google Search.
Updated on · All glossary terms