"Blocked by robots.txt": what it means and how to fix it
Your robots.txt file stops Google from visiting the page.
By Bruno Amorim · ·
What it means
A Disallow rule in your robots.txt file matches this page, so Google does not visit it. Google can still list a blocked page if other pages link to it, but without its content.
The fix, in short: Change or remove the Disallow rule that matches the page. If you blocked the page on purpose, take it out of your sitemap.
Which of your pages have this problem? Find out for free.
Get my free reportCommon causes
- A
Disallowrule blocks more pages than you meant to. - A
robots.txtfile from building the site stayed on after launch.
How to fix it
- If you want the page on Google, change the
robots.txtrule so it doesn't block this page. - If the page should stay off Google, take it out of your sitemap.
When you can ignore it
The page is an admin, search, or filter page that you blocked on purpose. Take it out of your sitemap.
If "Validate fix" failed
Validation fails here when the rule still matches the pages. Google generally keeps a copy of your robots.txt for up to 24 hours, so wait a day after you change it. Check the file Google read in Settings > robots.txt in Search Console. See what each validation result means.
Fixes by platform
On Shopify
Shopify's default robots.txt blocks admin, cart, checkout, account, and order pages, plus filtered (/collections/*+*) and sorted (sort_by) collection pages. That is on purpose, and you can ignore those. If a real product or page is blocked, someone added a rule in robots.txt.liquid. Delete the template to go back to the default. More in Shopify store not showing on Google.
On WordPress
WordPress's own robots.txt blocks only /wp-admin/. Since WordPress 5.3, Discourage search engines adds noindex instead of a robots.txt block. If more is blocked, a real robots.txt file in your site's root folder, or a plugin, added the rule. More in WordPress site not showing on Google.
On Squarespace
Settings > Crawlers has a Block Search engine crawlers box. It adds Disallow: / to your robots.txt, which blocks the whole site. Clear it. Squarespace's other rules block pages such as /config and /search on purpose, and its help says to ignore them. More in Squarespace website not showing on Google.
On Blogger
Blogger serves its own robots.txt unless you turn on Enable custom robots.txt under Crawlers and indexing in Settings. Custom content replaces the default. Turn it off to go back to Blogger's file.
How to find these pages
In Search Console, open Indexing > Pages and click the "Blocked by robots.txt" row. Search Console shows up to 1,000 example pages. CrawlCoach checks up to 50,000 pages per site and sorts them by reason, so you know which pages to fix first.
Questions
Does robots.txt keep a page off Google?
No. Google says robots.txt is not a way to keep a page out of Google. A blocked page can still be listed, without a description, if other pages link to it. To keep a page off Google, let Google visit it and add noindex.
How do I find the rule that blocks a page?
Inspect the page in URL Inspection. Crawl allowed? says No when robots.txt blocks it. Then open Settings > robots.txt in Search Console to see the file Google read, and look for the Disallow line that matches the address.
Related guides
- "Last crawl: N/A" in URL Inspection: what it means
- Sitemap "Couldn't fetch" in Search Console: causes and fixes
- Bing "URLs that are robots.txt disallowed": find the rule and fix it
- Google-Extended: a robots.txt token, not a crawler
- "Crawl allowed? No: blocked by robots.txt": how to fix it
- Sitemap URL errors: URL not allowed, Path mismatch, and Invalid URL
- Why Bing can't find your website: 8 blockers people miss
- "Indexing allowed? No: 'noindex' detected": how to fix it
- The best Google index checker tools
Sources
- Search Console Help: Page indexing report
- Google Search Central: Introduction to robots.txt
- Google Search Central: How Google interprets robots.txt
- Search Console Help: URL Inspection tool
- Shopify Help: Editing robots.txt.liquid
- WordPress Developer Resources: do_robots()
- WordPress.org: Settings Reading screen
- Squarespace Help: Hiding your site from search results
- Squarespace Help: Understanding Google SEO emails and console errors
- Blogger Help: Manage your blog's settings
Which of your pages have this problem?
Get my free reportFree for up to 500 pages. No card needed. Sign in with the Google account you use for Search Console.