Bing robots.txt disallowed URLs: how to fix it

Q: Does Bing follow Crawl-delay?

Yes. Bing's Crawl Control help says a Crawl-delay in robots.txt always wins over the Crawl Control setting. A high delay can leave pages "Discovered but not crawled" for a long time.

Q: Why is a page blocked when my * group allows it?

If robots.txt has a group that names bingbot, Bingbot follows that group only. Rules in the * group no longer apply to it. Put every rule Bing needs in the bingbot group, or delete the group.

A robots.txt rule tells Bingbot not to crawl these pages. Find the line that matches, check whether you meant it, and fix it if not.

Where you see it

In Bing Webmaster Tools, open Site Explorer and apply the filter "URLs that are robots.txt disallowed". The robots.txt tester shows which line blocks a page.

What it means

Your robots.txt file asks Bingbot not to fetch these pages, and Bingbot follows it. Bing can't read pages it is not allowed to crawl.

Bing's guidelines say robots.txt controls crawling, not what Bing shows. Bing's help also says a blocked page can still show in results with no description. To keep a page off Bing, use noindex instead, and let Bingbot crawl the page so it can see the rule.

Common causes

Most likely first.

A Disallow rule written for a folder also matches pages you want, for example Disallow: /p also blocks /products/.
A User-agent: bingbot group exists. Bingbot then follows only that group and skips your * rules, including any Allow lines.
A staging site's Disallow: / went live with the site.
A plugin, theme, or platform setting wrote rules for you.
Blocked CSS or JavaScript files, so Bingbot can't render the page.

How to tell which cause applies

Open yourdomain.com/robots.txt in a browser. Read every group, not just User-agent: *.
In Bing Webmaster Tools, open the robots.txt tester. Enter the page under Test URL, keep Bingbot selected, and click Test. It shows the line that blocks it.
From Site Explorer, you can send a blocked address straight to the robots.txt tester.
If you use an SEO plugin or a site builder, check its robots.txt setting. It may rebuild the file on each save.

How to fix it

Delete or narrow the rule that blocks pages you want on Bing. End folder rules with a slash, for example Disallow: /private/.
If you keep a bingbot group, copy into it every rule from the * group that should apply to Bing too.
Allow the CSS and JavaScript files your pages need.
Upload the new file. In the robots.txt tester, click Fetch latest and test the page again.
Click Request indexing in URL Inspection for the pages that matter most.

When you can ignore it

Ignore it for pages you block on purpose: admin areas, carts, internal search, and endless filter combinations. Bing's guidelines suggest blocking crawl paths like these.

With CrawlCoach

On paid plans, CrawlCoach reads your robots.txt and flags rules that block Bingbot from your public pages on Google. It does the same for the ChatGPT, Claude, Perplexity, and Apple search crawlers. The fix names the crawler and the pages, and CrawlCoach checks the file again after you change it.

Questions

Does Bing follow `Crawl-delay`?

Yes. Bing's Crawl Control help says a Crawl-delay in robots.txt always wins over the Crawl Control setting. A high delay can leave pages "Discovered but not crawled" for a long time.

Why is a page blocked when my `*` group allows it?

If robots.txt has a group that names bingbot, Bingbot follows that group only. Rules in the * group no longer apply to it. Put every rule Bing needs in the bingbot group, or delete the group.

Will blocking a page in robots.txt remove it from Bing?

Not reliably. Bing's guidelines say robots.txt controls crawling, not what Bing shows. Use noindex to keep a page off Bing, and leave it crawlable so Bingbot sees the rule.

Related guides

"URLs with NOINDEX tag" in Bing
Bing blockers people miss
"Discovered but not crawled" in Bing
Bingbot (Microsoft's crawler)
"Blocked by robots.txt" (the Google Search Console reason)
"Indexed, though blocked by robots.txt" (the Google Search Console reason)

Sources

Checked on October 8, 2026. Vendors change their tools and docs, so check the source if something looks different.

Bing "URLs that are robots.txt disallowed": find the rule and fix it