robots.txt tester.
A robots.txt is the note a website leaves for bots - a sign on the door saying "come in" or "please keep out". Fetch or paste one and I'll translate it into plain English, then you can check any page.
is your robots.txt doing what you think?
robots.txt looks simple, but a few of its rules behave in ways that catch people out. These are the ones worth knowing before you trust a file to do what you meant.
does User-agent: * apply to every bot?
It is a fallback, not a global rule. Google calls the * line the global user agent, which makes it sound like it applies to everyone, but it does not. A crawler obeys the single most specific group that names it and ignores every other group, including *. So the moment a bot has its own group it stops reading * completely and inherits nothing from it. To bind a named bot to a shared rule, you have to repeat that rule inside the bot's own group.
User-agent: * Disallow: / User-agent: Googlebot Disallow: /tmp/
Here Googlebot has its own group, so it reads only that: it can crawl the whole site except /tmp/. It never sees the Disallow: / under *. Every other bot falls back to * and is blocked from everything.
which robots.txt rule wins when rules conflict?
When rules clash, the longest matching path wins, regardless of the order they appear in the file. If an Allow and a Disallow match the same number of characters, Allow wins. That is why Allow: /folder beats Disallow: /folder, and why a long, specific rule can carve an exception out of a broad one.
Disallow: /blog/ Allow: /blog/public/
/blog/public/my-post is allowed: the matching Allow path is longer (13 characters versus 6), so it wins and carves an open door out of the blocked folder. A plain /blog/draft is still blocked - only the Allow matches it.
should I block pages I don't want indexed?
Not with robots.txt. A Disallow stops a bot fetching a page, but it does not keep the URL out of search: Google can still index a disallowed URL, showing it with no description, if other pages link to it. To keep something out of results, allow crawling and add a noindex tag, or put it behind a login. And do not reach for both at once: a Disallow plus a noindex backfires, because the crawler is blocked from fetching the page and so never sees the noindex telling it to drop out.
which crawlers ignore the global user agent (*)?
A handful of Google's own crawlers skip * even when nothing names them: AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google (AdSense) and APIs-Google. A blanket Disallow: / under * will not stop them, so you have to write a group that names them. Google-Safety goes further and ignores robots.txt entirely. The logic: advertisers pay for those crawls, so Google will not let a catch-all quietly break them.
should you block query strings in robots.txt?
Disallow: /*? blocks every URL with a query string. Tidy in theory, but it can also block the CSS and JS your pages load with cache-busting parameters (style.css?v=3), which stops Google rendering the page properly. Be specific about the parameters you actually want gone.
does it help crawl budget?
On large sites, yes. Keeping bots out of low-value URLs (internal search, faceted filters, admin) leaves more crawl time for the pages that matter. On a small site it rarely moves the needle, Google will get round to you either way. For the bigger picture, see the guide to crawlability and robots.txt.
why do some bots ignore it entirely?
It is a voluntary protocol, not a lock. Reputable engines honour it; aggressive scrapers and some AI bots do not. robots.txt controls crawling, not access, so if a page must stay private, protect it properly rather than relying on a Disallow line.
symbol cheat sheet
*- wildcard matches any run of characters.
$- end anchor pins the match to the end of the URL.
#- comment everything after it on the line is ignored.
There are no other wildcards. ?, . and the rest are treated as literal characters. For every directive and edge case, see the robots.txt reference.
how accurate is this, and where are Google's docs?
This tester is a static checker that follows Google's interpretation of the rules: it explains a file and tests a URL against it, but it does not fetch the page itself or check live response codes, and other crawlers (Bing, Yandex) differ in places, which it flags where it matters. The matching logic mirrors Google's own open-source robots.txt parser, so the allow and disallow decisions line up with how Googlebot actually reads a file. Straight from the source: