Robots.txt Blocks Crawling, Not Indexing

Crawling is fetching a URL. Indexing is storing what was found so that it can appear in results. Robots.txt only speaks to the first. Google's documentation is blunt about it, calling the file "not a mechanism for keeping a web page out of Google".

A search engine can know a URL exists without ever fetching it, because other pages link to it, and the link text says something about what it holds. So a disallowed URL can still be listed. Google's wording is that the URL "can still appear in search results, but the search result won't have a description", since the description would have come from the content it wasn't allowed to read.

The standard agrees on the limits. RFC 9309, which made the Robots Exclusion Protocol an internet standard in 2022, says these rules "are not a form of access authorization". The file is public, and anyone can read it, which also means a list of secret paths in robots.txt is a list of where to look. Well-behaved crawlers follow it because they choose to.

A file is divided into groups, each starting with one or more User-agent lines. The mistake people make is assuming the groups stack, with User-agent: * as a base layer and named groups added on top. They don't. Google's specification says its crawlers use only the single group with the most specific match, and every other group is ignored. RFC 9309 falls back to * only "if no matching group exists".

Take a file with Disallow: /admin/ under User-agent: *, and then Disallow: /tmp/ under User-agent: Googlebot. Bingbot is blocked from /admin/. Googlebot is not, because it found a group naming it and stopped looking. The moment you add a group for one crawler, you have to repeat in it every rule that should still apply.

The order of groups doesn't matter, and neither does the case of the crawler's name. Two groups naming the same crawler are merged, but a named group is never merged with *. Some crawlers also answer to a broader token: Googlebot-Image follows a Googlebot group when it has none of its own, and Apple says Applebot does the same, so a group written only for Googlebot reaches Apple's crawler too. A few skip the wildcard altogether. Google says AdsBot-Google ignores *, so User-agent: * plus Disallow: / does not stop it, and it needs a group of its own.

A rule matches from the start of the path, and that catches people out in both directions. Disallow: /fish also blocks /fish.html, /fishing/ and /fish?id=1, but not /catfish or /desert/fish. Ending it with a slash, as /fish/, narrows it to what is inside that folder and no longer blocks /fish itself.

Paths are case-sensitive, so /Fish and /fish are different rules, while field names and crawler names are not. There are only two special characters. * stands for zero or more of any character, and $ at the end of a rule anchors it to the end of the URL. So /*.pdf$ matches /report.pdf but not /report.pdf?download=1. To block a PDF whatever follows it, leave the $ off.

The query string is part of what gets matched, which is how /*?sort= works, and also why it misses /shoes?color=red&sort=price, where sort isn't the first parameter. The online store preset in the tool adds /*&sort= for that reason.

Encoding is compared after normalizing. RFC 9309's own table treats /foo/bar/%62%61%7A and /foo/bar/baz as the same path, and /foo/bar/ツ as the same as its percent-encoded form. An encoded slash (%2F) stays different from a real one.

The table above holds a surprise. A missing robots.txt costs nothing, and a site with no restrictions to declare doesn't need one. A robots.txt that returns a server error is another matter, because Google stops crawling the whole site for the first twelve hours rather than risk fetching something it was told not to. Firewalls and bot protection are a common cause: they answer the request with an error, the site looks fine in a browser, and crawling quietly stops.

Two other limits are worth knowing. Google generally caches the file for up to 24 hours, so an edit doesn't apply the second you save it, and the cache can last longer when refreshing fails. It also reads only the first 500 KiB and ignores the rest, which real files seldom reach unless they are generated from a list of URLs.

Blocking everything with "AI" in its description is the common reflex, and it can cost you something you wanted. A site that objects to training can block GPTBot and ClaudeBot while still appearing as a cited source, because those companies run their search crawlers under other tokens. A site that wants neither has to list both kinds.

Google-Extended trips people up in the opposite way. It has no crawler behind it, and its documentation says it has no separate HTTP user agent. It is a token that tells Google how content fetched by its other crawlers may be used. Blocking it won't change your server logs and won't affect your rankings.

The fetchers triggered by a person are the ones this file controls least. When someone asks an assistant about your page and it goes to read it, OpenAI and Perplexity both say robots.txt may not apply. If a page really must not be read by those tools, it needs a login, not a line in a text file.

Test the pages that must stay crawlable, not only the ones you meant to block. A rule that blocks too little is usually harmless, while a rule that catches your product pages is what costs money, and you only find it by testing those product URLs.

Run each URL as each crawler you have a group for, because a group for Googlebot can give it different answers from the ones everything else gets. Then compare with what Google has. The robots.txt report in Search Console shows the version it last fetched, when it fetched it and any errors, which tells you whether your edit has reached it yet.