About this tool
The tester follows Google's rules: the group for the most specific matching user agent applies, the longest matching Allow or Disallow path wins, and Allow wins a tie. It also flags syntax problems such as rules outside a group or a missing sitemap line.
How to use it
- Open yoursite.com/robots.txt, copy everything and paste it into the first box.
- List the URLs or paths you want to check in the second box, one per line.
- Pick the crawler. Each URL shows Allowed or Blocked and the rule that decided it.
Questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, but a blocked URL can still be indexed if other pages link to it. Use a noindex tag on a crawlable page to keep it out of results.
What do * and $ mean in robots.txt?
* matches any sequence of characters and $ marks the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf.
Why test robots.txt by pasting it?
Browsers cannot read another site's robots.txt directly, so paste the file (open yoursite.com/robots.txt and copy it). This also lets you test changes before you upload them.
Last reviewed 16 October 2026.