Google-Extended: a robots.txt token, not a crawler
Google-Extended is a robots.txt token, not a crawler. It controls Gemini training and grounding. It does not affect Google Search.
By Bruno Amorim · ·
What it does
Google says Google-Extended is a standalone product token. It lets you manage whether content Google crawls from your site may be used to train future Gemini models, and for grounding in the Gemini app and Vertex AI.
Google says it has no separate user-agent string. Google's existing crawlers do the crawling. The token only controls how the content is used.
Google says Google-Extended does not affect your site's inclusion in Google Search, and is not a ranking signal.
User agent
None. Google says Google-Extended does not have a separate HTTP user-agent string. You will not see it in your logs.
robots.txt
- Google-Extended works only in robots.txt. Googlebot reads the rule. There is nothing to allow in a firewall.
User-agent: Google-Extended
Allow: /User-agent: Google-Extended
Disallow: /Allowed is the default. You only need an allow group if another rule blocks the crawler. A group that names a crawler replaces your User-agent: * rules for it, so copy into it any Disallow lines you still want, such as your admin pages.
What blocking it costs you
- Your content is not used to train future Gemini models, or for grounding in the Gemini app and Vertex AI, per Google.
- No effect on Google Search. Google says AI Overviews and AI Mode are part of Search, and Googlebot's robots.txt rules are the control for them.
How to verify real traffic
- There is no Google-Extended traffic to check. Requests come from Google's normal crawlers. Verify those as Google describes for Googlebot.
Firewalls and CDNs
- Never block Googlebot to opt out of Gemini training. That removes you from Google Search. Use the Google-Extended robots.txt rule instead.
With CrawlCoach
Google-Extended is your call, so CrawlCoach does not flag it. CrawlCoach watches what matters for Google Search: which of your pages Google indexed, why it skipped the rest, and the fix for each.
Questions
What is Google-Extended?
A robots.txt token. It tells Google whether pages it crawls may be used for Gemini training and grounding. It is not a separate crawler and has no user agent.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says AI features in Search are part of Search, and robots.txt rules for Googlebot are the control. To limit what Search shows from a page, Google points to nosnippet, data-nosnippet, max-snippet, or noindex.
Does Google-Extended affect my rankings?
No. Google says it does not affect inclusion in Google Search and is not a ranking signal.
Related guides
- All AI crawlers compared
- Applebot and Applebot-Extended (Siri, Spotlight, Safari search, and Apple model training)
- GPTBot (Model training)
- ClaudeBot (Model training)
- "Excluded by 'noindex' tag" (the Google Search Console reason)
- "Blocked by robots.txt" (the Google Search Console reason)
Sources
Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.
Check that AI search can reach your pages
Every paid plan checks your robots.txt for the ChatGPT, Claude, Perplexity, Bing, and Apple search crawlers, and tests whether a firewall turns away OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Start with a free report on Google: up to 500 pages.
Get my free reportFree for up to 500 pages. No card needed. Sign in with the Google account you use for Search Console.