How robots.txt rules are matched
Crawlers read the group of rules addressed to their own user agent and fall back to the User-agent: * group only when no specific group exists. Inside a group, the most specific rule wins: the Allow or Disallow line with the longest matching path. When an Allow and a Disallow rule are equally long, Google uses the Allow rule.
Patterns may use * to match any sequence of characters and $ to anchor the end of a URL, so Disallow: /*.pdf$ blocks PDF files but not /guide.pdf?download=1. This tester follows the same logic Google documents.
Common robots.txt mistakes
The most expensive mistake is a leftover Disallow: / from a staging site, which tells every crawler to stay away from the whole site. Another is blocking CSS and JavaScript folders, which stops Google from rendering pages the way users see them.
Remember that robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Controlling AI crawlers
AI companies publish their own user agents, such as GPTBot (OpenAI), ClaudeBot (Anthropic) and CCBot (Common Crawl). You can allow or block each one with its own User-agent group. Test the result here by choosing the crawler in the dropdown.
Frequently asked questions
Where must robots.txt live?
At the root of each host and protocol, for example https://www.example.com/robots.txt. A file on www does not apply to the non-www host or to subdomains.
What happens if robots.txt returns an error?
A 404 means crawlers assume everything is allowed. A 5xx server error makes Google treat the whole site as disallowed until the file is reachable again, so fix server errors quickly.
Does Google respect Crawl-delay?
No. Google ignores Crawl-delay. Bing and some other crawlers respect it.
Is there a size limit?
Google reads the first 500 KiB of a robots.txt file and ignores the rest.