AI crawlers: which to allow, and how
AI companies send different crawlers for search, for model training, and for pages a person asks about. Allow the search ones to show up in AI answers. Training is your choice.
Every AI crawler at a glance
What each vendor's own docs say. Click a name for the user agent, robots.txt lines, and IP ranges.
| Crawler | Company | What it is for | Follows robots.txt | To show up in AI answers |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search | Yes | AllowNeeded to show up in ChatGPT search answers |
| GPTBot | OpenAI | Model training | Yes | Your choiceDoes not affect ChatGPT search |
| ChatGPT-User | OpenAI | Visits a page a person asked for | May not apply, per OpenAI | AllowLets ChatGPT read your page when a person asks |
| ClaudeBot | Anthropic | Model training | Yes | Your choiceDoes not affect Claude's search results |
| Claude-SearchBot | Anthropic | Claude's search results | Yes | AllowBlocking it may hide you in Claude's search |
| Claude-User | Anthropic | Visits a page a person asked for | Yes, per Anthropic | AllowLets Claude read your page when a person asks |
| PerplexityBot | Perplexity | Perplexity search | Yes | AllowNeeded to show up in Perplexity's results |
| Perplexity-User | Perplexity | Visits a page a person asked for | Generally ignores it, per Perplexity | AllowLets Perplexity read and link your page for a person |
| Google-Extended | Controls Gemini training and grounding | A robots.txt token, not a crawler | Your choiceDoes not affect Google Search or AI Overviews | |
| Applebot and Applebot-Extended | Apple | Siri, Spotlight, Safari search, and Apple model training | Yes (no Crawl-delay) | AllowAllow Applebot. Applebot-Extended is your choice. |
| Bingbot | Microsoft | Bing search and Microsoft Copilot | Yes | AllowNeeded for Bing and Microsoft Copilot |
A robots.txt that keeps you in AI answers
Every crawler is allowed unless a rule says otherwise. So a business that wants to show up in AI answers needs no lines for the search crawlers. This block opts out of model training only. Add it to the end of your robots.txt, and keep your own rules above it.
# AI search crawlers: allowed by default.
# Opt out of AI model training.
# Delete any group you are fine with.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /Each group applies only to the crawler it names. A crawler with its own group ignores your User-agent: * rules, so copy any Disallow lines you still want into its group. Do not block Googlebot or bingbot to opt out of AI. That removes you from Google or Bing search, and from the AI answers built on them.
robots.txt allows it, but the firewall does not
A firewall, CDN, or bot protection rule can refuse a crawler before it reads robots.txt. "Block AI bots" settings are the usual cause. OpenAI and Perplexity both ask you to allow their search crawlers' published IP ranges, not only the robots.txt rule. If you use Cloudflare, open Security Settings > Configure AI bot policies and set Search to Allow (do not block).
With CrawlCoach
Every paid plan checks your robots.txt for the search crawlers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, and Applebot. It flags any that are blocked from your public pages on Google. It also asks for one public page as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, and as a plain crawler. When only the AI crawler is refused, the fix list says a firewall rule likely blocks it, and how to allow it. Training crawlers stay your choice, so CrawlCoach does not flag them. CrawlCoach reads your pages with its own crawler, CrawlCoachBot. For Bing itself, see the Bing guides.
Search crawlers
Allow these to show up in AI search answers.
- OAI-SearchBotOAI-SearchBot finds pages for ChatGPT search. Block it and ChatGPT search won't show your pages in its answers. Here is how to allow it.
- Claude-SearchBotClaude-SearchBot crawls pages to improve Claude's search results. Anthropic says blocking it may reduce your visibility there.
- PerplexityBotPerplexityBot finds and links pages in Perplexity's search results. Perplexity says it does not crawl for model training.
- Applebot and Applebot-ExtendedApplebot crawls for Siri, Spotlight, and Safari. Applebot-Extended is a robots.txt token that opts you out of Apple's model training.
- BingbotBingbot crawls for Bing search, and Bing's guidelines say Copilot uses the same crawling and index. Block it and you lose both.
Training crawlers and tokens
Blocking these does not remove you from AI search. It is your choice.
- GPTBotGPTBot collects pages that may train OpenAI's models. Blocking it does not remove you from ChatGPT search. Here are the exact lines.
- ClaudeBotClaudeBot collects pages that could help train Anthropic's models. Blocking it is your choice. It does not remove you from Claude's search.
- Google-ExtendedGoogle-Extended is a robots.txt token, not a crawler. It controls Gemini training and grounding. It does not affect Google Search.
Fetchers a person triggers
They load one page when someone asks an AI assistant about it.
- ChatGPT-UserChatGPT-User visits a page when a person in ChatGPT asks for it. It is not a search crawler, and OpenAI says robots.txt may not apply.
- Claude-UserClaude-User visits pages when a person asks Claude a question. Blocking it may reduce your visibility in searches people run in Claude.
- Perplexity-UserPerplexity-User loads a page when a person asks Perplexity a question. Perplexity says it generally ignores robots.txt, since a person asked.