Skip to content
crawlcoach.
Get my free report

AI crawler checker

See if ChatGPT, Claude, and Perplexity can reach your page. We read your robots.txt for each AI crawler, then request the page as each AI search crawler to catch a firewall that turns it away.

A public page, such as your home page or a blog post.

What each result means

  • Blocked by robots.txt. A Disallow rule covers this page for that crawler. The result names the line. Remove it, or add an Allow for the crawler.
  • Refused by your firewall or bot rules. A plain request loaded the page, but the same request with the crawler's name got 401, 403, 406, 429, 451, 503, or a Cloudflare challenge. Something in front of your site turns that crawler away, even when robots.txt allows it.
  • Allowed, and the page loaded. Nothing refused the crawler's name. A firewall may still check the real crawler's IP address, which a test from our address can't copy. So this rules out a block by name, not every block.
  • Allowed by robots.txt. For Bingbot and Applebot we read robots.txt only. Firewalls that verify these crawlers check their IP address, so a request from us with their name would look fake and prove nothing.
  • Unknown. robots.txt answered with an error, sent a web page, or the page didn't load for a plain request. Fix that first, then check again.

The page text check reads the HTML your server sends, without running JavaScript. If the main text appears only after scripts run, crawlers that don't run JavaScript see an almost empty page.

Search crawlers vs training crawlers

AI companies use separate crawlers for search and for training. Search crawlers find pages to show and link in answers. Training crawlers collect pages to train models. Blocking a training crawler doesn't remove you from AI search.

CompanySearch crawlerTraining crawler or token
OpenAIOAI-SearchBotGPTBot
AnthropicClaude-SearchBotClaudeBot
PerplexityPerplexityBotNone
MicrosoftBingbotNone
AppleApplebotApplebot-Extended
GoogleGooglebotGoogle-Extended

Google is the odd one. AI Overviews and AI Mode are part of Google Search, and Googlebot's rules control them. Google-Extended is only a robots.txt token for Gemini training and grounding. It doesn't change Search. Never block Googlebot to opt out of AI. That takes you off Google.

Allow AI search, refuse training

To show up in AI answers but keep your content out of model training, add the groups below. Search crawlers are allowed by default, so they need no lines.

Add to robots.txt
# AI search crawlers: allowed by default.

# Opt out of AI model training.
# Delete any group you are fine with.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

One catch: a group that names a crawler replaces your User-agent: * rules for it. If you add a group for a search crawler, copy in the Disallow lines you still want, such as your cart or admin pages. Test the result in the robots.txt tester before you upload it.

Cloudflare's AI bot setting

If robots.txt allows a crawler but the request as that crawler is refused, look at your CDN's bot settings. In Cloudflare, open Security Settings > Configure AI bot policies. It has three categories: Search, Agent, and Training. Each can be set to Block (on all pages), Block on pages with ads, or Allow (do not block).

  • Set Search to Allow (do not block) if you want to show up in AI answers.
  • Cloudflare says that from September 15, 2026, blocking Training also blocks crawlers that do both search and training.
  • New domains from that date block Training and Agent bots on pages with ads by default. Search stays allowed.

Run this check again after any change. Other CDNs and security plugins have similar bot rules.

On every paid plan, each fix list runs these checks across your site. It reads robots.txt for the five AI search crawlers on up to 50 public pages per host, tests the firewall on your home page, and flags pages with search traffic whose text loads only with JavaScript.

Questions

How do I check if ChatGPT can crawl my site?

Enter a page above. The OAI-SearchBot row shows whether robots.txt allows ChatGPT's search crawler, and whether a request with its name loaded the page. GPTBot is a separate crawler for OpenAI's model training, listed under training crawlers.

Is my site blocked from AI?

Two things can block AI crawlers: a robots.txt rule, or a firewall or bot rule. This check tests both for ChatGPT, Claude, and Perplexity search. For Bingbot and Applebot it reads robots.txt only, because firewalls that verify those crawlers check their IP address.

If I block GPTBot, will my site disappear from ChatGPT?

No. OpenAI treats each crawler on its own. GPTBot is for training. ChatGPT search uses OAI-SearchBot. You can block GPTBot and still allow OAI-SearchBot.

Does the robots.txt result prove that AI will cite my page?

No. It shows whether robots.txt allows the crawler, and whether a firewall refuses a request with its name. Each AI picks the pages it shows, and no setting guarantees a citation.

Related guides

Sources

Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.