How to generate a robots.txt for your site

1. Pick the preset

   "Preset" picks between `allow-all` (let all bots crawl everything), `disallow-all` (block every bot from every URL — useful for staging), or `disallow-admin` (allow most, block `/admin` and `/private`). The choice writes the `User-agent: *` block.

2. Sitemap URL (optional)

   "Sitemap URL" appends a `Sitemap:` line at the end, pointing at your XML sitemap. Search engines that find this read it without you having to submit elsewhere. Use an absolute URL (`https://example.com/sitemap.xml`).

3. Press Run

   Result returns a single `robotsTxt` field. Save it as a literal file named `robots.txt` at your site's root — it must be reachable at `https://example.com/robots.txt` (not `/blog/robots.txt`).

4. What robots.txt isn't

   It's a request, not enforcement. Well-behaved bots (Googlebot, Bingbot) respect it; malicious crawlers ignore it. For real access control use auth, IP blocks, or rate limits — and never list secrets in robots.txt (you'd be advertising them).