Robots.txt Generator
Build a robots.txt
file with your own rules: a different set of Allow/Disallow
paths per crawler if you need it, plus your sitemap URLs. Free
tool, nothing you enter is stored.
How to use
-
1
Add a rule group for each crawler you want to target, or leave the one default
*group as-is to cover every crawler. -
2
List the paths to Disallow (and, optionally, Allow) for that group, then add your Sitemap URLs if you have any.
-
3
Click Generate robots.txt, then Copy or Download the result and upload it to your site's own root as
/robots.txt.
What is a robots.txt file?
robots.txt
is a small text file that tells well-behaved web crawlers
(search engines, AI crawlers, SEO tools) which parts of your
site they may or may not request. It has to live at your
domain's own root
(https://yoursite.com/robots.txt)
for crawlers to find it; this tool generates the
file's content, but uploading it there is a separate step you
do yourself.
How do I use it?
Save the generated text as a file literally named
robots.txt
and upload it to your site's own root directory. Each
User-agent
block above targets one crawler (or
*
for every crawler that doesn't have its own more specific
block); its
Disallow
lines list paths that crawler shouldn't request, and
Allow
lines carve out exceptions within an otherwise-disallowed
area. Any
Sitemap
URLs you add point crawlers at your sitemap directly. That's
useful alongside, not instead of, submitting it to each
search engine's own tools.
robots.txt
is advisory, not a security mechanism. It only works
because well-behaved crawlers choose to respect it. A
disallowed path is still often reachable by anyone who has
(or guesses) its direct URL, and a disallow rule can even
make a path's existence more discoverable, not less, since
the file itself is public. Never rely on it to hide genuinely
sensitive content; use real access control (authentication, a
firewall rule, or simply not linking to it) for anything that
actually needs to stay private.
FAQ
Is this tool free?
Yes, no login required either.
Do you store what I generate?
No. Nothing about a generation (your rules, the output, or that it happened at all) is saved anywhere. The file is shown once, in the same response, and gone.
Can I have different rules for different crawlers?
Yes. Add one rule group per crawler (e.g. a group for GPTBot and a separate one for *) and each gets its own User-agent block in the output.
Where do I put the generated file?
Save it as a file literally named robots.txt and upload it to your site's own root, so it's reachable at https://yoursite.com/robots.txt.
Does robots.txt keep my pages out of Google?
Not reliably. It's a crawling request, not a de-indexing mechanism, so a disallowed page can still end up indexed if it's linked to from somewhere else. To actually keep a page out of search results, use a noindex meta tag or HTTP header instead.
Is robots.txt a security feature?
No. It only works because well-behaved crawlers choose to respect it, and the file itself is public, so a disallow rule can even make a path's existence more discoverable, not less. Never rely on it to hide anything genuinely sensitive.
Related tools
All toolsXML Sitemap Generator
Enter your URL and get a clean XML sitemap from a real crawl of up to 500 pages, with a reason for each page that is left out.
Open toolCanonical URL Checker
Enter a URL and check its canonical tag: self-referencing, scheme/host match, and whether the target is actually reachable.
Open toolDead Link Checker
Find broken outbound links on a page or across a site, and check inbound links this app has recorded.
Open tool