ClaudeBot: what Anthropic's crawler does and how to block it
ClaudeBot collects pages that could help train Anthropic's models. Blocking it is your choice. It does not remove you from Claude's search.
By Bruno Amorim · ·
What it does
Anthropic says ClaudeBot collects web content that could contribute to training its generative AI models.
When a site blocks ClaudeBot, Anthropic says the site's future content should be left out of its training datasets.
Anthropic runs two other agents: Claude-SearchBot for search, and Claude-User for pages a person asks Claude to read. Each has its own robots.txt setting.
User agent
Anthropic's help page names the robots.txt token ClaudeBot but does not publish the full user-agent string. Match on ClaudeBot in the user agent, and check the IP against Anthropic's list.
robots.txt
- Anthropic says its bots honor robots.txt and do not try to get past CAPTCHAs.
- Anthropic also supports the
Crawl-delayline, which most crawlers ignore. Use it to slow ClaudeBot instead of blocking it. - Add the rule to robots.txt on every subdomain you want to cover.
User-agent: ClaudeBot
Allow: /User-agent: ClaudeBot
Disallow: /Allowed is the default. You only need an allow group if another rule blocks the crawler. A group that names a crawler replaces your User-agent: * rules for it, so copy into it any Disallow lines you still want, such as your admin pages.
What blocking it costs you
- Your future pages are left out of Anthropic's training data.
- Claude's search results are a separate setting, Claude-SearchBot, so blocking ClaudeBot does not hide you there.
How to verify real traffic
- Anthropic publishes one IP list for its bots. Anthropic says a crawler with a source IP on that list comes from Anthropic.
- Anthropic warns that blocking its IPs instead of using robots.txt may not work as an opt-out, because then it can't read your robots.txt.
- Anthropic crawler IP addresses (JSON)
Firewalls and CDNs
robots.txt is only half of it. A firewall, CDN, or bot rule can turn a crawler away before it reads robots.txt.
- If a firewall rule blocks "Anthropic" or "Claude" by name, it may also catch Claude-SearchBot and Claude-User. Block only
ClaudeBotif training is all you want to stop. - If you use Cloudflare, open Security Settings > Configure AI bot policies. Set Search to Allow (do not block). Cloudflare's docs say that since September 15, 2026, blocking Training also blocks crawlers that do both search and training.
With CrawlCoach
Blocking ClaudeBot is your call, so CrawlCoach does not flag it. CrawlCoach does flag robots.txt rules and firewall refusals that hit Claude-SearchBot, the crawler Claude's search uses.
Questions
What is the ClaudeBot user agent?
Anthropic's help page gives the token ClaudeBot but no full user-agent string. Look for ClaudeBot in the user agent, and confirm the IP is on Anthropic's published list.
How do I block ClaudeBot?
Add User-agent: ClaudeBot and Disallow: / to robots.txt on each subdomain. Anthropic says robots.txt is the way to opt out. IP blocks may not work reliably.
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic supports Crawl-delay. For example: User-agent: ClaudeBot then Crawl-delay: 1.
Related guides
- All AI crawlers compared
- Claude-SearchBot (Claude's search results)
- Claude-User (Visits a page a person asked for)
- GPTBot (Model training)
- Google-Extended (Controls Gemini training and grounding)
Sources
Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.
Check that AI search can reach your pages
Every paid plan checks your robots.txt for the ChatGPT, Claude, Perplexity, Bing, and Apple search crawlers, and tests whether a firewall turns away OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Start with a free report on Google: up to 500 pages.
Get my free reportFree for up to 500 pages. No card needed. Sign in with the Google account you use for Search Console.