---
title: "robots.txt generator — free, runs in your browser"
description: "Build a robots.txt with crawler rules, AI bot access and a sitemap line. Free, no sign-up, and nothing you enter leaves your browser."
url: https://atoneplace.net/tools/robots-txt-generator/
site: "At One Place"
---

# robots.txt generator

Build a robots.txt with crawler rules, AI bot access and a sitemap line.

## How it works

Produces a robots.txt from a few choices, including whether to allow AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot.

The most important thing to understand about robots.txt is what it does not do. It requests that a crawler not fetch a URL; it does not prevent indexing. A page blocked in robots.txt can still appear in results if other pages link to it, because the crawler never fetches it and therefore never sees a noindex tag. To keep a page out of an index you need a noindex meta tag or X-Robots-Tag header — and the page must be crawlable for that to be read.

That combination catches people constantly: blocking a URL in robots.txt and adding noindex to the page achieves nothing, because the noindex is never seen.

robots.txt is public and advisory. Anyone can read it, and badly-behaved crawlers ignore it entirely, so it is not a security measure and should never list paths you want kept private.

> This tool runs entirely in your browser. Nothing you enter is sent to a server, logged or stored, and the page keeps working with the network disconnected.

## Common questions

### Does robots.txt stop a page being indexed?

No. It asks crawlers not to fetch the page. A blocked page can still be indexed from external links, and its noindex tag will never be read because the page is never fetched.

### How do I actually keep a page out of Google?

A noindex meta tag or X-Robots-Tag header, on a page that is crawlable. Blocking it in robots.txt prevents the noindex from being seen.

### Should I allow AI crawlers?

It depends on your goals. Allowing them makes your content quotable in AI answers; blocking them keeps it out of training and retrieval. It is a business decision, not a technical one.

## Related pages

- [Password generator](https://atoneplace.net/tools/password-generator/)
- [Merge PDF files](https://atoneplace.net/tools/pdf-merge/)
- [QR code generator](https://atoneplace.net/tools/qr-code-generator/)
- [Image resizer](https://atoneplace.net/tools/image-resizer/)
- [Split and extract PDF pages](https://atoneplace.net/tools/pdf-split/)
- [Random number generator](https://atoneplace.net/tools/random-number-generator/)
- [Random picker and team generator](https://atoneplace.net/tools/random-picker/)
- [Image crop, rotate and flip](https://atoneplace.net/tools/image-crop-rotate/)
- [Lorem ipsum generator](https://atoneplace.net/tools/lorem-ipsum-generator/)
- [Meta tag generator](https://atoneplace.net/tools/meta-tag-generator/)

## Sources

- Typical range compiled from manufacturer specifications — At One Place (/about/)