What is Robots.txt Generator?
A robots.txt file sits at the root of your site and tells search engine crawlers which parts they may visit. It is a public request rather than a security control, so it should never be the only protection for private pages.
This generator turns the rules into a correctly formatted file. Add a group for each crawler, list the paths to allow or block, and point to your sitemap so search engines find your pages faster.
How to use
- Enter your site URL and choose a default rule (allow all or block all).
- Add groups for specific crawlers, with the paths to allow or disallow.
- Add your sitemap URL, then copy the result or download robots.txt and upload it to your site root.
Frequently asked questions
Where does robots.txt go?
It must be served from the root of the domain, for example https://example.com/robots.txt. A file in a subfolder is ignored by search engines.
Does robots.txt hide pages from Google?
No. It stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it. Use a noindex meta tag or password protection to keep a page out of search.
Should I block AI crawlers?
That is your choice. Many sites add rules for AI training bots such as GPTBot or CCBot. Compliance depends on each crawler honouring robots.txt.
Does Crawl-delay work for Google?
Google ignores Crawl-delay. Bing and some other crawlers respect it.