Something wrong with this tool?
About Robots.txt Generator Online
This tool builds a robots.txt file for your website, telling search engine crawlers and bots which paths they should and shouldn't index. Specify rules per user-agent (Googlebot, Bingbot, generic *), list allow/disallow paths, add sitemap URLs, and the tool outputs a ready-to-deploy robots.txt.
A correct robots.txt prevents wasteful crawling of admin areas, search result pages, and private content, while explicitly inviting crawlers to your sitemap. Missing or misconfigured robots.txt can lead to bloated indexes, slow site crawling, or accidentally blocking public pages.
Place the resulting file at the root of your domain (https://example.com/robots.txt). The file is just a hint to well-behaved bots — it doesn't enforce access control, so don't rely on it to hide sensitive content from determined scrapers.
How to use this tool
How to generate a robots.txt for your site
Pick the preset
"Preset" picks between `allow-all` (let all bots crawl everything), `disallow-all` (block every bot from every URL — useful for staging), or `disallow-admin` (allow most, block `/admin` and `/private`). The choice writes the `User-agent: *` block.
Sitemap URL (optional)
"Sitemap URL" appends a `Sitemap:` line at the end, pointing at your XML sitemap. Search engines that find this read it without you having to submit elsewhere. Use an absolute URL (`https://example.com/sitemap.xml`).
Press Run
Result returns a single `robotsTxt` field. Save it as a literal file named `robots.txt` at your site's root — it must be reachable at `https://example.com/robots.txt` (not `/blog/robots.txt`).
What robots.txt isn't
It's a request, not enforcement. Well-behaved bots (Googlebot, Bingbot) respect it; malicious crawlers ignore it. For real access control use auth, IP blocks, or rate limits — and never list secrets in robots.txt (you'd be advertising them).