Claude-User user agent: what it is and robots.txt

Claude-User visits pages when a person asks Claude a question. Blocking it may reduce your visibility in searches people run in Claude.

What it does

Anthropic says Claude-User supports Claude users. When people ask Claude questions, it may visit websites with this agent.

It is not a crawler that roams the web on its own schedule. It fetches pages for a person's request.

User agent

Anthropic's help page names the token Claude-User but does not publish the full user-agent string. Match on Claude-User, and check the IP against Anthropic's list.

robots.txt

Anthropic says Claude-User lets site owners control which sites these user-initiated requests can reach, and that its bots honor robots.txt.
That differs from OpenAI and Perplexity, which say robots.txt may not apply to their user-triggered fetchers.

To allow it

User-agent: Claude-User
Allow: /

To block it

User-agent: Claude-User
Disallow: /

Allowed is the default. You only need an allow group if another rule blocks the crawler. A group that names a crawler replaces your User-agent: * rules for it, so copy into it any Disallow lines you still want, such as your admin pages.

What blocking it costs you

Anthropic says blocking Claude-User stops it from fetching your content for a user's question, which may reduce your visibility in user-directed web search.

How to verify real traffic

Check the request's IP against Anthropic's published list.
Anthropic crawler IP addresses (JSON)

Firewalls and CDNs

robots.txt is only half of it. A firewall, CDN, or bot rule can turn a crawler away before it reads robots.txt.

Bot rules that block all AI agents catch Claude-User. Allow it if you want people to be able to ask Claude about your pages.
If you use Cloudflare, open Security Settings > Configure AI bot policies. Set Search to Allow (do not block). Cloudflare's docs say that since September 15, 2026, blocking Training also blocks crawlers that do both search and training.

With CrawlCoach

CrawlCoach checks Claude-SearchBot, Anthropic's search crawler, in robots.txt and through your firewall. It leaves the person-triggered fetchers of each vendor out of its checks.

Questions

Does Claude-User follow robots.txt?

Anthropic says its bots honor robots.txt, and that Claude-User lets you control which sites user requests can reach. So a Disallow for Claude-User should stop it.

Should I block Claude-User?

Usually not. It only visits when a person asks Claude about something. Blocking it means Claude can't read your page for that person.

Related guides

All AI crawlers compared
Claude-SearchBot (Claude's search results)
ClaudeBot (Model training)
ChatGPT-User (Visits a page a person asked for)
Perplexity-User (Visits a page a person asked for)

Sources

Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.

Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?

Claude-User: what it is and whether to block it