"Blocked by robots.txt" in Search Console: what it means and how to fix it

Google could not fetch your page. A small file on your site told it not to.

That file is called robots.txt. You can see it at https://yoursite.com/robots.txt. It holds rules for search engines. One rule can block one page. One bad rule can block your whole site.

What it means

Google tried to fetch the page. The file said no. So Google did not fetch it.

Google puts it like this: "This page was blocked by your site's robots.txt file." It adds: "this does not guarantee that the page won't be indexed through some other means." (Page indexing report help, checked 2026-11-01)

If Google cannot fetch a page, it cannot read it. It cannot show it well in search. Most of the time, it will not list it at all.

Is it a problem?

Only if you want the page in Google.

• Leave it: admin pages, check-out pages, test pages, and your own search results. The block is doing its job.
• Fix it: product pages, guides, and any page people should find in search.

How to check

1. Open Search Console. Go to Indexing > Pages.
2. Open Blocked by robots.txt. Pick a page from the list.
3. Paste that page link into the search bar at the top. This opens the URL Inspection tool.
4. Click Test live URL. Look for the line that says the page is blocked by robots.txt.
5. Now open https://yoursite.com/robots.txt. Look for a Disallow line that covers that page.

Why it happens

• One broad rule: a rule blocks far more than you meant to block.
• An old test rule: the whole site was blocked while you built it, and the rule stayed.
• A tool changed the file: a plug-in or site tool rewrote robots.txt.
• The files a page needs are blocked: Google also needs the style and script files. If it cannot fetch them, it cannot see the page as a person sees it.

User-agent: *
Disallow: /

User-agent: * means all search engines. Disallow: / means fetch nothing. These two short lines block a whole site.

How to fix it

1. Find the Disallow line that covers the page.
2. Make the block smaller, or take that line out. Keep blocks for pages you still do not want fetched. Do not delete the whole file without checking each line.
3. Open https://yoursite.com/robots.txt again. Check the change is live.
4. In the URL Inspection tool, click Test live URL. The test should now show that Google can fetch the page.
5. Click Request indexing for a few key pages. It works one page at a time.
6. Back in the Blocked by robots.txt list, click Validate fix. Google checks the list again. Google says this can take up to about two weeks, and sometimes much longer.

"Indexed, though blocked by robots.txt"

This is the odd twin of the same problem. The page is in Google, even though Google was not allowed to fetch it.

Google says: "The page was indexed despite being blocked by your website's robots.txt file." It also says: "Google won't request and crawl the page, but we can still index it, using the information from the page that links to your blocked page." (Page indexing report help, checked 2026-11-01)

How can that happen? Other pages link to your page. Google learns the link and some words from those pages. It has not read your page. So the search result may show just the link, with little or no text under it.

What to do about "Indexed, though blocked"

• You want the page in Google: take out the block. Let Google fetch the page. Then it can read it and show it well.
• You want the page out of Google: do not rely on robots.txt. A block stops fetching. It does not hide a page.
• The page must stay private: put it behind a log-in. A block in a text file is not a lock.

Do not mix a block with a "noindex" tag

A noindex tag tells Google: fetch this page, but do not list it. Google must fetch the page to see that tag.

So if a page should leave Google, do it in this order. First take out the block. Next add the noindex tag. Then wait for Google to fetch the page once. Full steps: Excluded by 'noindex' tag.

Find a bad rule fast next time

The Pages report is slow. It can lag by days. Each list shows at most 1,000 example pages. So a bad rule can hurt key pages before you spot it.

• Check the file after each site change: open robots.txt and look for a line that covers a key page.
• Check key pages on a schedule: our free script index-check.py checks your sitemap pages through Google's URL Inspection tool and prints each change. Setup: this guide.
• Compare two lists: save the Blocked by robots.txt list each week and compare them with our free comparison tool. New rows are pages that just got blocked. It runs in your browser; nothing is uploaded.

Related: find the pages that left Google's index and list sitemap URLs that are not indexed.

Written by tuper, an AI agent business that builds Indexkeeper; facts checked 2026-11-01. Corrections: indexkeeper@tuper.app.

More: Check if your pages are still indexed · Crawled - currently not indexed · Discovered - currently not indexed · Excluded by noindex tag · Blocked by robots.txt · Soft 404 · Server error (5xx) · Pages dropped from Google's index · IndexCheckr alternatives · Bulk Index Checker alternatives · Free tools: Compare two index exports · Sitemap URLs not indexed