“Blocked by robots.txt” in Google Search Console

What the status means

A rule in your robots.txt tells Googlebot not to request this URL, so Google never reads the page. That's all the status says. It reports a crawling decision you (or a plugin, a platform or a past developer) made, and Google follows it.

Two things it doesn't mean. It isn't an error from Google's side, and it doesn't guarantee the page stays out of search. Google's help says a blocked page can still be indexed in rare cases, if Google finds information about it without loading it. That case has its own status: Indexed, though blocked by robots.txt.

Is it a problem?

How to find the rule

1. Open Indexing → Pages → Blocked by robots.txt and sort the examples by section. A pattern, such as everything under /blog/, points to one rule.
2. Test a URL. Paste your file and the URL into our robots.txt tester. It shows which line decides and how the result changes for Googlebot and other crawlers.
3. Confirm in URL Inspection. Search Console's robots.txt report (under Settings) shows whether Google fetched the file and when, but it doesn't test individual URLs. For that, Google points to the URL Inspection tool.

Check the file on the exact host and protocol. Each subdomain has its own robots.txt: the rules at www.example.com don't apply to shop.example.com.

Why it happens

1. A staging rule reached production. Disallow: / is the classic one, often after a launch or a migration.
2. A rule that matches more than you meant. Rules match URL prefixes, so Disallow: /blog also blocks /blog-news/ and /blogging-tips/. Use /blog/ if you mean the folder.
3. A wildcard. Disallow: /*? blocks every URL with a parameter, including ones you may want crawled.
4. Plugin or platform defaults. An SEO plugin, security plugin or platform writes rules you didn't add by hand.
5. A specific group overriding the general one. Googlebot follows only the most specific group that matches it. If the file has User-agent: Googlebot, the rules under User-agent: * no longer apply to it.

When rules conflict inside a group, Google uses the one with the longest path. If they're equally specific, it uses the least restrictive one, so Allow wins the tie.

How to fix it

Delete or narrow the rule, then check the result in the tester before you publish it.

WordPress

• A physical robots.txt in the site root replaces the one WordPress generates. Look there first.
• Edit it in Yoast (Tools → File editor) or Rank Math (General Settings → Edit robots.txt).

Shopify

• The default file blocks cart, checkout, admin, internal search and filtered collection URLs. That's by design.
• Custom rules live in the robots.txt.liquid theme template. Only edit it if you know which rule you need to change.

Wix and Squarespace

• Wix has a robots.txt editor in its SEO settings.
• Squarespace doesn't let you edit the file, so its blocked URLs are platform defaults. Use the page-level setting to hide pages from search.

Next.js and headless sites

• In the App Router, the file comes from app/robots.ts or public/robots.txt. Check that the value isn't tied to an environment variable still set to staging.

After publishing

• Google generally caches robots.txt for up to 24 hours. After a critical fix, use Request a recrawl in the robots.txt report to speed it up.
• Then run Validate fix for the issue. Google says validation typically takes up to about two weeks.

If you wanted the page out of Google

Blocking crawling is the wrong tool. Google's documentation says robots.txt “is not a mechanism for keeping a web page out of Google”.

Remove the rule and add a noindex directive. Google has to crawl the page to see it, so if you keep both, the noindex is never read.

What else to check

If robots.txt itself returns a server error, the effect is much wider than one URL. Google's documentation says it stops crawling the site for the first 12 hours, then uses the last good copy for up to 30 days. A 4xx response is treated as if the file didn't exist, which means no restrictions at all.