Claude-SearchBot: Anthropic's search crawler, and why to allow it

What it does

Anthropic says Claude-SearchBot crawls the web to improve search result quality for Claude users, and to make search answers more relevant and accurate.

It is separate from ClaudeBot, which collects training data. You can allow one and block the other.

User agent

Anthropic's help page names the token Claude-SearchBot but does not publish the full user-agent string. Match on Claude-SearchBot, and check the IP against Anthropic's list.

robots.txt

• Anthropic says its bots honor robots.txt, and support Crawl-delay.

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-SearchBot
Disallow: /

Allowed is the default. You only need an allow group if another rule blocks the crawler. A group that names a crawler replaces your User-agent: * rules for it, so copy into it any Disallow lines you still want, such as your admin pages.

What blocking it costs you

• Anthropic says blocking Claude-SearchBot stops it from indexing your content for search, which may reduce your visibility and accuracy in Claude's search results.

How to verify real traffic

• Check the request's IP against Anthropic's published list.
• Anthropic crawler IP addresses (JSON)

Firewalls and CDNs

robots.txt is only half of it. A firewall, CDN, or bot rule can turn a crawler away before it reads robots.txt.

• A rule that blocks "Claude" or "Anthropic" by name catches Claude-SearchBot too. If you only want to stop training, block ClaudeBot alone.
• If you use Cloudflare, open Security Settings > Configure AI bot policies. Set Search to Allow (do not block). Cloudflare's docs say that since September 15, 2026, blocking Training also blocks crawlers that do both search and training.

With CrawlCoach

On every paid plan, CrawlCoach reads your robots.txt and flags rules that block Claude-SearchBot from your public pages on Google. It also asks for one public page as Claude-SearchBot and as a plain crawler, and tells you when a firewall likely refuses Claude-SearchBot.

Questions

Related guides

• All AI crawlers compared
• ClaudeBot (Model training)
• Claude-User (Visits a page a person asked for)
• OAI-SearchBot (ChatGPT search)
• PerplexityBot (Perplexity search)

Sources

Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.

• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?
• Cloudflare docs: Block AI bots and AI bot policies