robots.txt generator
Build a robots.txt with crawler rules, AI bot access and a sitemap line.
How it works
Produces a robots.txt from a few choices, including whether to allow AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot.
The most important thing to understand about robots.txt is what it does not do. It requests that a crawler not fetch a URL; it does not prevent indexing. A page blocked in robots.txt can still appear in results if other pages link to it, because the crawler never fetches it and therefore never sees a noindex tag. To keep a page out of an index you need a noindex meta tag or X-Robots-Tag header — and the page must be crawlable for that to be read.
That combination catches people constantly: blocking a URL in robots.txt and adding noindex to the page achieves nothing, because the noindex is never seen.
robots.txt is public and advisory. Anyone can read it, and badly-behaved crawlers ignore it entirely, so it is not a security measure and should never list paths you want kept private.
This tool runs entirely in your browser. Nothing you enter is sent to a server, logged or stored, and the page keeps working with the network disconnected.
Common questions
- Does robots.txt stop a page being indexed?
- No. It asks crawlers not to fetch the page. A blocked page can still be indexed from external links, and its noindex tag will never be read because the page is never fetched.
- How do I actually keep a page out of Google?
- A noindex meta tag or X-Robots-Tag header, on a page that is crawlable. Blocking it in robots.txt prevents the noindex from being seen.
- Should I allow AI crawlers?
- It depends on your goals. Allowing them makes your content quotable in AI answers; blocking them keeps it out of training and retrieval. It is a business decision, not a technical one.
Related pages
- Password generator Generate strong random passwords in your browser, with the entropy shown.
- Merge PDF files Combine several PDFs into one, in the order you choose.
- QR code generator Make a QR code for a link, text, WiFi network or contact details, and download it as SVG.
- Image resizer Resize an image to exact dimensions or a percentage, in your browser.
- Split and extract PDF pages Pull out a page range, split into single pages, or delete pages.
- Random number generator Random numbers in any range, plus dice, coin flips and a shuffled pick from a list.
- Random picker and team generator Pick a winner from a list, split into teams, or generate a random order.
- Image crop, rotate and flip Crop to an aspect ratio, rotate in 90° steps, or mirror an image.
- Lorem ipsum generator Generate placeholder text by paragraphs, sentences or words.
- Meta tag generator Generate title, description, canonical, Open Graph and Twitter Card tags.
Sources
- Typical range compiled from manufacturer specifications — At One Place