Google-Extended
Definition
Google-Extended is a token to use in robots.txt, introduced by Google in 2023: no crawler by that name visits sites. It lets you say whether the pages crawled by Googlebot can be used to improve Gemini models and to ground their answers in the Gemini app and Google's APIs.
The trap many sites fall into: blocking Google-Extended doesn't remove a site from AI Overviews or AI Mode, which are part of Google Search and go through Googlebot. Apple offers a similar mechanism with Applebot-Extended.
Example
“User-agent: Google-Extended” then “Disallow: /”: the site stays in Google and in AI Overviews, but asks that its content not be used for Gemini models.
What it means for your AI visibility
Before you block Google-Extended, know what you're protecting and what you're giving up: less presence in what Gemini knows about you, with no effect on your visibility in Google Search.
Related terms
- robots.txtrobots.txt is the file that tells crawlers, including AI crawlers, which parts of a site they're allowed to visit.
- GeminiGemini is Google's family of AI models, and the conversational assistant of the same name, which replaced Bard in 2024.
- Google AI OverviewsAI Overviews are the AI-generated summaries Google shows at the top of its search results, with links to their sources.
- AI crawlerAn AI crawler is a bot that browses the web on behalf of an AI company, to train its models or feed its answers.
Updated on · All glossary terms