"Crawl allowed? No: blocked by robots.txt": how to fix it

What it means

Before Google visits a page, it reads your robots.txt file. A Disallow rule matches this page, so Google stayed away.

Blocking the crawl does not always keep a page out of search. If other pages link to it, Google can still list the address, without its content. That shows as Indexed, though blocked by robots.txt.

Google can't see a noindex tag on a page it can't crawl. Google's help says Indexing allowed? then always says Yes, even if the page has noindex.

Where it shows: Page indexing > Crawl > Crawl allowed?

Common causes

1. A Disallow rule is wider than you meant. For example, Disallow: /blog also blocks /blog-news.
2. A rule from a test or staging site stayed on after launch, such as Disallow: /.
3. A plugin or platform setting added rules for search, tag, or filter pages.
4. A wildcard rule, such as Disallow: /*?, blocks every address with a query string.

How to check which cause applies

1. Open https://your-domain/robots.txt in your browser. Read the rules under User-agent: * and User-agent: Googlebot.
2. Open Settings > robots.txt in Search Console to see the version Google last fetched. Google's help says this report is only for Domain properties and URL-prefix properties at the top of a site.
3. Find the rule that matches the page's path. Google uses the most specific rule, which is the longest one. If two rules are equal, the less strict one wins.

How to fix it

1. If the page should be on Google, change or remove the rule that matches it.
2. Click Test live URL. Crawl allowed? should now say Yes.
3. Click Request indexing. For an urgent change, you can also click Request a recrawl for the file in the robots.txt report.
4. If the page should stay off Google, remove it from your sitemap. To keep it out of search for sure, allow the crawl and add noindex instead.

When you can ignore it

Admin, cart, internal search, and filter pages are often blocked on purpose. That is fine. Just keep them out of your sitemap.

With CrawlCoach

CrawlCoach reads Google's robots.txt result for each page it checks. On paid plans, its fix list flags pages in your sitemap that robots.txt blocks, because the sitemap and the rule disagree. Google makes the final call on what it indexes.

Related Page indexing reasons

• Blocked by robots.txt: Your robots.txt file stops Google from visiting the page.
• Indexed, though blocked by robots.txt: Google lists the page because of links, but robots.txt stops Google from reading it.

Questions

Sources

Checked on October 8, 2026. Google can change its tools and wording after that date.

• Search Console Help: URL Inspection tool
• Google Search Central: How Google interprets robots.txt
• Search Console Help: robots.txt report