GPTBot: user agent, robots.txt, and how to block it
GPTBot collects pages that may train OpenAI's models. Blocking it does not remove you from ChatGPT search. Here are the exact lines.
By Bruno Amorim · ·
What it does
OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models.
Disallowing GPTBot tells OpenAI your content should not be used for that training.
GPTBot is separate from OAI-SearchBot. Blocking GPTBot does not keep you out of ChatGPT search.
User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotOpenAI calls this an example: the version number may change. When it fetches robots.txt, it may add a robots.txt marker to the string. Match on GPTBot.
robots.txt
- OpenAI lists GPTBot as a robots.txt tag. A
Disallowrule for GPTBot is how you opt out of training.
User-agent: GPTBot
Allow: /User-agent: GPTBot
Disallow: /Allowed is the default. You only need an allow group if another rule blocks the crawler. A group that names a crawler replaces your User-agent: * rules for it, so copy into it any Disallow lines you still want, such as your admin pages.
What blocking it costs you
- Nothing in ChatGPT search. OpenAI treats GPTBot and OAI-SearchBot as separate settings.
- Your future pages are not used to train OpenAI's models. That is the point of blocking it.
How to verify real traffic
- Check the request's IP against OpenAI's published list for GPTBot.
- GPTBot IP ranges (JSON)
Firewalls and CDNs
robots.txt is only half of it. A firewall, CDN, or bot rule can turn a crawler away before it reads robots.txt.
- If you block GPTBot in a firewall, match the user agent
GPTBotexactly. A broad rule like "block OpenAI" or "block AI bots" can catch OAI-SearchBot too, and cost you ChatGPT search. - If you use Cloudflare, open Security Settings > Configure AI bot policies. Set Search to Allow (do not block). Cloudflare's docs say that since September 15, 2026, blocking Training also blocks crawlers that do both search and training.
With CrawlCoach
Blocking GPTBot is your call, so CrawlCoach does not flag it. CrawlCoach does flag robots.txt rules and firewall refusals that hit OAI-SearchBot, the crawler ChatGPT search needs.
Questions
How do I block GPTBot in robots.txt?
Add two lines: User-agent: GPTBot and Disallow: /. Leave OAI-SearchBot allowed if you want to show up in ChatGPT search.
Does blocking GPTBot hurt my SEO?
Not on Google or Bing. GPTBot is OpenAI's training crawler. It has nothing to do with Google or Bing search. It also does not decide whether ChatGPT search shows you.
Does GPTBot respect robots.txt?
OpenAI says disallowing GPTBot tells it your content should not be used in training. OpenAI publishes GPTBot's IP ranges, so you can check that requests claiming to be GPTBot are real.
Related guides
- All AI crawlers compared
- OAI-SearchBot (ChatGPT search)
- ChatGPT-User (Visits a page a person asked for)
- ClaudeBot (Model training)
- Google-Extended (Controls Gemini training and grounding)
Sources
Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.
Check that AI search can reach your pages
Every paid plan checks your robots.txt for the ChatGPT, Claude, Perplexity, Bing, and Apple search crawlers, and tests whether a firewall turns away OAI-SearchBot, Claude-SearchBot, or PerplexityBot. Start with a free report on Google: up to 500 pages.
Get my free reportFree for up to 500 pages. No card needed. Sign in with the Google account you use for Search Console.