Question 1

Why is a URL blocked for Bingbot but allowed for Googlebot?

Accepted Answer

Because a crawler obeys one group only. If your file has a User-agent: Googlebot group, Googlebot reads that group and skips everything under User-agent: *, while Bingbot, with no group of its own, reads the * rules. Pick each crawler in the tester to see which group it lands on. To apply a rule to both, repeat it in both groups.

Question 2

Will blocking a page remove it from Google?

Accepted Answer

No, and it can stop the removal from working. Disallow prevents crawling, so a blocked URL that other sites link to can still be listed, without a description. Google also can't see a noindex tag on a page it isn't allowed to fetch. To get a page out of results, allow crawling and add noindex, then block it later if you still want to.

Question 3

Which rule wins when Allow and Disallow both match?

Accepted Answer

The one with the longer path. With Disallow: /shop/ and Allow: /shop/sale/, the page /shop/sale/shoes is allowed because the Allow rule is longer. The order of the lines makes no difference. When the two are the same length, Allow wins. The tester names the winning rule, so you can see this on your own file.

Question 4

What do * and $ do in a path?

Accepted Answer

* stands for any run of characters, including none, so /*.pdf matches a PDF in any folder. $ marks the end of the URL, so /*.pdf$ matches /files/report.pdf but not /files/report.pdf?download=1. Rules match from the start of the path, which means /fish also covers /fish.html and /fishing/ unless you end it with a slash or a $.

Question 5

Should I block AI crawlers?

Accepted Answer

It depends on what you want from them, which is why the tool sorts them by job. GPTBot, ClaudeBot and Google-Extended relate to training models, and blocking Google-Extended has no effect on Google Search. OAI-SearchBot, Claude-SearchBot and PerplexityBot feed AI search answers, so blocking them can cost you citations there. Fetches made when a person asks a question, such as Perplexity-User and ChatGPT-User, may not follow these rules at all. The Block AI training only option covers the common choice of opting out of training while staying citable.

Question 6

What is the difference between Default, Allow and Block?

Accepted Answer

Default writes nothing for that crawler, so it follows your User-agent: * rules. Block gives it a group with Disallow: /. Allow gives it a group with Allow: /, and because a crawler reads only the group that names it, Allow opens the whole site to it, including any path you blocked for everyone else. That is what you want when the rest of the file blocks everything and you are letting a few crawlers in, and rarely what you want otherwise.

Question 7

Why does blocking Googlebot also block Applebot?

Accepted Answer

Apple says that when a file mentions Googlebot and not Applebot, Applebot follows the Googlebot instructions. Googlebot-Image, Googlebot-Video and Googlebot-News fall back on a Googlebot group in the same way. The Who can crawl this site panel shows these knock-on effects. To keep one of them out of it, set that crawler to Allow or give it rules of its own.

Question 8

Does the tool read Crawl-delay?

Accepted Answer

The file check flags it instead of acting on it. Google ignores Crawl-delay, so it has no effect on Googlebot, although some crawlers such as ClaudeBot do read it. The tester answers whether a URL may be fetched, which Crawl-delay doesn't change.

Question 9

Can I test the live file on my site?

Accepted Answer

Open yourdomain.com/robots.txt in a browser, copy the text and paste it under Test my own file. Search Console's robots.txt report is worth checking as well, because it shows the version Google last fetched and any errors it hit fetching it.

Question 10

My file allows a page but Search Console says it is blocked. Why?

Accepted Answer

Google caches robots.txt, generally for up to 24 hours, so a recent edit may not have reached it yet. Also check that you tested the right host, because https://www.example.com and https://example.com each use their own file, as does every subdomain.

Robots.txt Generator & Tester

What kind of site is this?

What about AI crawlers?

Any pages to keep crawlers out of? · optional

Where is your sitemap? · optional

Crawlers one by one

Rules for one crawler

Crawl delay for all crawlers

Your robots.txt

Who can crawl this site

Test URLs against this file

About Robots.txt Generator & Tester

How to use Robots.txt Generator & Tester

Frequently Asked Questions