Robots.txt Generator
Create a valid robots.txt: allow or block crawlers, disallow paths with wildcards, add Allow exceptions, Crawl-delay, sitemap URLs and block AI training bots.
Paths start with /. * matches any characters and $ marks the end of the URL.
Exceptions inside disallowed folders; the most specific rule wins.
Respected by Bing and Yandex; Google ignores it.
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /*?sessionid= Sitemap: https://example.com/sitemap.xml
What is Robots.txt Generator?
This generator builds a robots.txt file following RFC 9309. Choose whether crawlers may access the site, list paths to disallow (with * and $ wildcards) and Allow exceptions, add an optional Crawl-delay and your sitemap URLs, and optionally add a group that blocks AI training and AI search crawlers such as GPTBot, ClaudeBot, Google-Extended, CCBot and PerplexityBot.
How to use Robots.txt Generator
- Choose the default rule: allow crawling except listed paths, or block the whole site.
- Enter Disallow paths and Allow exceptions, one per line, each starting with /.
- Add Crawl-delay if needed and your sitemap URLs.
- Tick "Also block AI … crawlers" if you want to opt out of AI training and AI search.
- Copy or download robots.txt and upload it to your site root.
Example
Allow all except /admin/, /cart/, /*?sessionid= · sitemap https://example.com/sitemap.xml
User-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /*?sessionid= Sitemap: https://example.com/sitemap.xml
When to use Robots.txt Generator
Use it to stop crawlers from wasting time on admin, cart, search or parameter URLs, to point them at your sitemap, or to opt out of AI crawlers. robots.txt is advisory: it does not hide pages and does not remove indexed URLs – use noindex or authentication for that – and malicious bots ignore it.
How Robots.txt Generator works
The tool writes a User-agent: * group with your Allow and Disallow lines (an empty Disallow: when nothing is blocked, which keeps the group valid), then an optional group listing the AI user agents with Disallow: /, then Sitemap lines. Lines that do not start with / or * are flagged because crawlers ignore them. Crawl-delay is only added when it is a number.
Common Use Cases
- Block /admin/ and /cart/ while allowing the rest of a shop.
- Block an entire staging site from crawlers.
- Opt out of GPTBot, ClaudeBot and Google-Extended.
- Declare multiple sitemap files.
FAQs
Does Disallow remove a page from Google?
No. A blocked URL can still be indexed from links, just without content. Use a noindex meta tag (and allow crawling) to keep it out of results.
Does Google respect Crawl-delay?
No. Google ignores it; Bing and Yandex respect it.
Will blocking Google-Extended remove me from Google Search?
No. Google-Extended only controls use of your content for Gemini and Vertex AI training; Googlebot still crawls for search.