Robots.txt Tester

How to use the robots.txt tester

1. Fetch the live file: enter the site or any page on it; the tester loads /robots.txt from that host. Or paste rules to test a change before you publish it.
2. Enter the URL or path to test. Paths are case-sensitive, and the query string counts: /shop/?sort=price is a different path from /shop/.
3. Read the table. Each crawler is Allowed or Blocked, with the line that decided and the user-agent group it used. The deciding line for Googlebot is highlighted in the file.

How Google reads robots.txt

The tester follows Google's documented rules:

• Groups. A group starts with one or more User-agent lines followed by rules. A crawler uses the group for the most specific user agent that matches it; if none matches, the * group. Several groups for the same user agent are merged. A crawler with its own group ignores the * group completely, which is the most common surprise: rules you wrote for everyone don't apply to Googlebot once you add a Googlebot group.
• Longest match wins. Within the group, the rule whose path matches the most characters decides. When an Allow and a Disallow are equally long, Allow wins.
• Wildcards. * matches any run of characters; $ marks the end of the URL. Disallow: /*.pdf$ blocks /guide.pdf but not /guide.pdf?download=1.
• Supported fields. Google reads user-agent, allow, disallow and sitemap. It ignores crawl-delay and anything else; the tester lists those lines.
• Size. Google reads the first 500 KiB of the file.

Some Google crawlers obey more than one token. Googlebot-Image, for example, follows a Googlebot-Image group if there is one and otherwise the Googlebot group.

When robots.txt itself fails

A robots.txt that errors out is worse than having none at all. Search Console's robots.txt report shows when Google last fetched your file and whether it could parse it.

Blocking crawling is not blocking indexing

Disallow stops crawlers from fetching a page. It doesn't keep the URL out of Google: a blocked page that other sites link to can still be indexed, usually with no description, and Search Console reports “Indexed, though blocked by robots.txt”. To keep a page out of search results, let Google crawl it and add noindex. Our redirect & indexability checker checks both for any URL.

AI crawlers

AI companies crawl with their own user agents, so you can decide about them separately from search:

Blocking one of them doesn't affect Googlebot. If being cited by ChatGPT or Perplexity matters to your business, check that their search crawlers are allowed; it is one of the first things we look at in AI search optimization.