Research · October 29, 2026
We read 4,763 WordPress sites. 15% block an AI bot.
The top WordPress sites on the web, read on October 29: robots.txt, sitemaps, the homepage, and llms.txt. WordPress sites block AI bots 12 times as often as Shopify stores, mostly the bots that train models. One in five now serves an llms.txt file.
- 14.9%
- block at least one AI bot
- Shopify stores: 1.2%
- 4.6%
- block an AI search bot
- ChatGPT, Claude, or Perplexity search
- 20.9%
- serve an llms.txt file
- 997 sites
- 16.8%
- of the top sites run WordPress
- 5,126 of 30,457 homepages
Most block training, not search.
682 of 4,588 sites block at least one AI bot in robots.txt. 464 of them block GPTBot, which collects training data for OpenAI, and still allow OAI-SearchBot, which ChatGPT search uses.
OpenAI says the two are separate: blocking GPTBot does not keep a site out of ChatGPT search, and sites that block OAI-SearchBot are not shown in ChatGPT search answers. Google says Google-Extended does not affect Google Search.
What each AI crawler does| Bot | Used for | Sites that block it |
|---|---|---|
| OAI-SearchBot | ChatGPT search | 124 (2.7%) |
| ChatGPT-User | ChatGPT visits a page for a user | 216 (4.7%) |
| PerplexityBot | Perplexity search | 207 (4.5%) |
| Claude-SearchBot | Claude search | 97 (2.1%) |
| Bingbot | Bing and Copilot | 20 (0.4%) |
| GPTBot | OpenAI model training | 588 (12.8%) |
| ClaudeBot | Anthropic model training | 525 (11.4%) |
| Google-Extended | Gemini training and grounding | 468 (10.2%) |
| Applebot-Extended | Apple model training | 472 (10.3%) |
| CCBot | Common Crawl | 600 (13.1%) |
997 sites serve an llms.txt file.
llms.txt is a proposed file, from llmstxt.org, that lists a site's main pages in Markdown for AI tools. SEO plugins now write it for you: 309 of the 1,082 files we found say Yoast SEO generated them.
Check your llms.txt| The file says it was generated by | Files |
|---|---|
| Yoast SEO | 309 |
| Rank Math | 77 |
| All in One SEO | 65 |
| Does not say | 631 |
Half run Yoast.
Each plugin writes a comment into the homepage or the sitemap. At least 49.8% of sites carry Yoast's. On 36% we found no plugin's mark: those sites use a less common plugin or write their own tags.
| SEO plugin | Sites |
|---|---|
| Yoast SEO | 49.8% |
| Rank Math | 8.5% |
| All in One SEO | 3.6% |
| The SEO Framework | 1.5% |
| Other plugins | 0.5% |
| None we could detect | 36% |
Archive pages fill many sitemaps.
Category, tag, and author archives are pages too, and each one in a sitemap is a URL Google may crawl. On the 793 sites where we could count every post and tag file, the median listed 664 posts and 115 tag archives.
- 49.1%
- list category archives
- 2,165 of 4,407 sitemaps
- 29.2%
- list author archives
- 1,285 sitemaps
- 27.2%
- list tag archives
- 1,198 sitemaps
- 18.2%
- list more tag archives than posts
- 144 of 793 fully counted
- 7.5%
- have no sitemap where we looked
- robots.txt, sitemap_index.xml, wp-sitemap.xml, sitemap.xml
- 11.6%
- still use WordPress's default robots.txt
- Three lines, plus a sitemap
Check your own site
01
Open yoursite.com/robots.txt.
Look for OAI-SearchBot, PerplexityBot, or Claude-SearchBot followed by Disallow: /. Those block AI search. Bingbot covers Bing and Copilot. Change the file in your SEO plugin's robots.txt editor.
Test your robots.txt for AI bots02
Open Search Console, then Sitemaps.
Check the sitemap your SEO plugin writes is submitted. Then open Pages to see which archive URLs Google left out.
03
Open yoursite.com/llms.txt.
It is optional. If your SEO plugin writes one, check that it lists your main pages.
Check your llms.txt
How we did it
Read the Shopify store studyWhich sites
We read the Tranco list of top sites (list ZJ8PG) in rank order, from the top to rank 47,688. 30,457 homepages loaded; 5,126 ran WordPress, by the WordPress REST API link or wp-content paths in the page.
Domains that redirect to the same site count once, which leaves 4,763. The list ranks some hosting and ad domains, so a few sites are not ordinary websites.
What we read
For each site, on 2026-10-29: robots.txt, the homepage, the sitemap and up to 30 of its files, and /llms.txt. Percentages use the sites where that file could be read; each table says its base.
We read public files only. They show what a site tells Google and AI crawlers, not what Google has indexed.
Our crawler
Requests came from CrawlCoachBot. We skipped the 34 WordPress sites whose robots.txt disallows it.
See which of your pages Google left out.
CrawlCoach's free report reads your Search Console: up to 500 pages, which ones Google left out and why, and the 3 to fix first.
Get my free reportFree for up to 500 pages. No card needed. Sign in with the Google account you use for Search Console.