About this tool
robots.txt tells well-behaved crawlers which paths to skip. It is not access control: blocked URLs can still be indexed if other sites link to them.
Writing a robots.txt file that does what you expect
A robots.txt file sits at the root of your website and tells well-behaved crawlers which parts of the site they should not visit. It is useful for keeping search engines away from admin areas, shopping carts, internal search results and endless filtered pages that waste crawl effort without adding value. This generator builds the file from simple choices and includes a link to your sitemap so crawlers can find your important pages quickly.
It is important to understand what robots.txt does not do. It is a request, not a lock. Reputable search engines follow it, but anyone can still open a blocked URL directly, and badly behaved bots may ignore it entirely. Blocking a page in robots.txt also does not guarantee that it disappears from Google; if other sites link to it, the address can still appear in results without a description. To keep a page out of search results, allow it to be crawled and add a noindex tag instead. For private content, use proper login protection.
The generator also offers a quick way to block known AI training and answer crawlers, for site owners who prefer not to have their content collected for those purposes. Policies and crawler names change over time, so review the list occasionally. Be very careful with broad rules: a single line blocking everything can remove a whole site from search if published by mistake. After uploading the file, open your domain followed by /robots.txt in a browser to confirm it is live, and check it in Google Search Console.
How to use it
- Adjust the settings.
- Check the live preview.
- Copy the generated code into your page or stylesheet.
Questions
Will robots.txt remove a page from Google?
No. To remove a page, let Google crawl it and add a noindex meta tag, or remove the page.
How do I block AI training crawlers?
Choose 'Block AI crawlers' to add disallow rules for GPTBot, ClaudeBot, Google-Extended, CCBot and similar bots.
Last reviewed 16 October 2026.