Skip to content
crawlcoach.
Get my free report

robots.txt tester

Check whether robots.txt lets Googlebot, Bingbot, and AI crawlers reach a page. You get Allowed or Blocked for each crawler, the exact line that decided it, and why.

We download its live robots.txt, such as example.com/robots.txt.

One address or path per line, up to 10. Leave it empty to test the home page.

Crawlers

A user agent name, such as Yandex or AhrefsBot, or a full user agent string.

How to read the results

Each page gets one line per crawler: Allowed or Blocked, and the rule that decided it. Select the rule to jump to its line in the file. Open Why to see which group the crawler used and why that rule won over the others.

No rule matches means the crawler may fetch the page. robots.txt only blocks what a rule names. The file check lists problems with the file itself: its HTTP status, its size, and lines Google skips.

How Google reads robots.txt

Google documents how it reads the file, based on the RFC 9309 standard. This tester uses the same rules.

  1. One group per crawler. A crawler uses the group that names it, such as User-agent: Googlebot. Only when no group names it does it use User-agent: *. The two never combine. Several groups for the same crawler merge into one.
  2. The longest matching rule wins. Allow: /shop/sale beats Disallow: /shop for /shop/sale/shoes, because it is longer. When an Allow and a Disallow are the same length, Allow wins.
  3. * and $ are the only wildcards. * matches any run of characters. $ marks the end of the address. Disallow: /*.pdf$ blocks every PDF, but not /guide.pdf?v=2.
  4. Paths are case-sensitive. Disallow: /fish blocks /fish.html but not /Fish.html. Field names and crawler names are not case-sensitive.
  5. Rules match from the start of the path. Disallow: /fish blocks /fishheads too, because it is a prefix. It does not block /catfish. Use /fish/ to block only the folder.
  6. Only four fields count. Google reads user-agent, allow, disallow, and sitemap. It ignores Crawl-delay, and it stopped reading noindex lines in 2019.
  7. 500 KiB at most. Google ignores everything after the first 500 KiB of the file.

Some crawlers borrow another group. Google lists Googlebot as a second token for Googlebot-Image, so with no Googlebot-Image group, it follows the Googlebot group. Apple says Applebot follows Googlebot's rules when robots.txt doesn't mention Applebot.

What the HTTP status of robots.txt does

robots.txt answersWhat Google does
200Reads the file and follows its rules.
3xx redirectFollows at least five redirects, then treats the file as a 404.
404, 410, other 4xxActs as if there is no file: nothing is blocked. 429 is the exception.
5xx, timeoutStops crawling the site for the first 12 hours and keeps retrying. Then uses the last good copy for up to 30 days.

That last row is the one that hurts. A robots.txt that errors can stop Google from crawling every page, even pages no rule blocks.

Mistakes that block a whole site

These are the ones that take a site off Google's crawl. The tester flags each of them.

  • A leftover Disallow: /. Staging and new sites often ship with it. One slash blocks every page. An empty Disallow: blocks nothing, so the two look alike and do the opposite.
  • A group that drops your * rules. Adding User-agent: Googlebot with one rule means Googlebot ignores every rule under User-agent: *. Copy the rules you still want into the new group.
  • A server error on /robots.txt. A firewall or CDN that refuses the file with a 5xx, or a server that times out, stops Google's crawl of the whole site.
  • A web page instead of a text file. Some servers answer a missing robots.txt with the home page and a 200 status. Google skips the HTML lines, so the rules you think you set do nothing.
  • Blocked CSS and JavaScript. Rules like Disallow: /*.js$ stop Google from rendering your pages the way visitors see them.

robots.txt blocks crawling, not indexing

A Disallow rule stops Google from reading a page. It doesn't keep the address off Google. If other sites link to it, Google can index the bare address with no snippet. Search Console then reports Indexed, though blocked by robots.txt.

To keep a page off Google, do the opposite: allow crawling and add a noindex tag, so Google can see it. If the page should be on Google and Search Console says Blocked by robots.txt, remove the rule this tester points to.

On every paid plan, CrawlCoach checks your robots.txt and up to 50 of your most important pages every day. If a rule starts blocking Googlebot from one of them, or robots.txt stops answering, you get an email.

What happened to Google's robots.txt Tester

Google retired the old robots.txt Tester at the end of 2023. Its replacement is the robots.txt report in Search Console, under Settings > robots.txt. It works for Domain properties and for URL-prefix properties at the root of a site.

The report shows, for the top 20 hosts on your site:

  • the robots.txt file Google last fetched, and earlier versions,
  • its fetch status, such as Fetched or Not Fetched - Not found (404),
  • the size of the file and any issues Google found.

It can also ask Google to fetch the file again: click the three dots next to it and choose Request a recrawl. What it can't do is test a URL. For one page on your own site, run URL Inspection and read the Crawl allowed? line. To test a draft file, many pages at once, AI crawlers, or a site you don't own, use the tester above.

Questions

Is there still a robots.txt tester in Google Search Console?

No. Google retired the robots.txt Tester at the end of 2023 and replaced it with the robots.txt report, under Settings > robots.txt. The report shows the file Google last fetched and any errors, but it can't test a URL. To test one page, use the Crawl allowed? line in URL Inspection, or the tester on this page.

How do I check if robots.txt is blocking Googlebot?

Enter your site and the page address above, and keep Googlebot ticked. The result shows Allowed or Blocked, with the exact Disallow line that decided it. In Search Console, URL Inspection shows the same answer as Crawl allowed? Yes or No.

Can I validate a robots.txt before I upload it?

Yes. Choose Paste a robots.txt, paste your draft, and list the pages you care about. The tester shows what each crawler may fetch and flags lines Google ignores. Nothing you paste is saved.

How long does Google take to see a robots.txt change?

Google caches robots.txt for up to 24 hours. To speed it up after you unblock important pages, open Settings > robots.txt in Search Console, click the three dots next to the file, and choose Request a recrawl.

Related guides

Sources

Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.