Skip to content

"Blocked by robots.txt": what it means and how to fix it

Your robots.txt file stops Google from visiting the page.

A setting blocks it

By Bruno Amorim · ·

What it means

A Disallow rule in your robots.txt file matches this page, so Google does not visit it. Google can still list a blocked page if other pages link to it, but without its content.

The fix, in short: Change or remove the Disallow rule that matches the page. If you blocked the page on purpose, take it out of your sitemap.

Which of your pages have this problem? Find out for free.

Get my free report

Common causes

  • A Disallow rule blocks more pages than you meant to.
  • A robots.txt file from building the site stayed on after launch.

How to fix it

  • If you want the page on Google, change the robots.txt rule so it doesn't block this page.
  • If the page should stay off Google, take it out of your sitemap.

When you can ignore it

The page is an admin, search, or filter page that you blocked on purpose. Take it out of your sitemap.

If "Validate fix" failed

Validation fails here when the rule still matches the pages. Google generally keeps a copy of your robots.txt for up to 24 hours, so wait a day after you change it. Check the file Google read in Settings > robots.txt in Search Console. See what each validation result means.

Fixes by platform

On Shopify

Shopify's default robots.txt blocks admin, cart, checkout, account, and order pages, plus filtered (/collections/*+*) and sorted (sort_by) collection pages. That is on purpose, and you can ignore those. If a real product or page is blocked, someone added a rule in robots.txt.liquid. Delete the template to go back to the default. More in Shopify store not showing on Google.

On WordPress

WordPress's own robots.txt blocks only /wp-admin/. Since WordPress 5.3, Discourage search engines adds noindex instead of a robots.txt block. If more is blocked, a real robots.txt file in your site's root folder, or a plugin, added the rule. More in WordPress site not showing on Google.

On Squarespace

Settings > Crawlers has a Block Search engine crawlers box. It adds Disallow: / to your robots.txt, which blocks the whole site. Clear it. Squarespace's other rules block pages such as /config and /search on purpose, and its help says to ignore them. More in Squarespace website not showing on Google.

On Blogger

Blogger serves its own robots.txt unless you turn on Enable custom robots.txt under Crawlers and indexing in Settings. Custom content replaces the default. Turn it off to go back to Blogger's file.

How to find these pages

In Search Console, open Indexing > Pages and click the "Blocked by robots.txt" row. Search Console shows up to 1,000 example pages. CrawlCoach checks up to 50,000 pages per site and sorts them by reason, so you know which pages to fix first.

Questions

Does robots.txt keep a page off Google?

No. Google says robots.txt is not a way to keep a page out of Google. A blocked page can still be listed, without a description, if other pages link to it. To keep a page off Google, let Google visit it and add noindex.

How do I find the rule that blocks a page?

Inspect the page in URL Inspection. Crawl allowed? says No when robots.txt blocks it. Then open Settings > robots.txt in Search Console to see the file Google read, and look for the Disallow line that matches the address.

Related guides

Sources

Which of your pages have this problem?

Get my free report

Free for up to 500 pages. No card needed. Sign in with the Google account you use for Search Console.

Related problems