robots.txt Generator
Free tool · by Daniel HaketTell search engines what to crawl. Choose your rules, add the paths to block and your sitemap, and this builds a valid robots.txt to copy straight into your site's root.
What this free tool is great for: a quick, one-off job with no signup — it runs entirely in your browser, so nothing leaves your device and there's nothing to manage.
Its honest limit: it checks or generates one thing in your browser — it won't track rankings over time, monitor changes across many pages, or optimise at scale.
What robots.txt is and what it controls
A robots.txt file is a small text file that lives at the root of your site and tells search-engine crawlers where they may and may not go. Before a well-behaved bot crawls your pages, it checks this file for instructions. This tool builds a valid robots.txt from simple choices — which bots to address, which paths to keep them out of, and where your sitemap lives — so you get correct syntax without hand-writing directives and risking a typo that either does nothing or, worse, quietly blocks half your site.
The distinction that everything depends on: crawling is not indexing
This is the single most misunderstood point in all of technical SEO, so it's worth stating plainly. robots.txt controls crawling — whether a bot is allowed to fetch a URL. It does not control indexing — whether that URL can appear in search results. Those are different things, and confusing them causes real damage. If you Disallow a page in robots.txt but other sites link to it, Google can still index the URL — it just won't have crawled the content, so it shows a bare link with no description. Blocking crawling does not hide a page from search. To actually keep a page out of the index, you need a noindex directive on the page itself, which means the page must remain crawlable so the bot can see the noindex. Block it in robots.txt and you've ironically prevented Google from ever seeing the very instruction that would remove it.
The syntax, in plain terms
The format is small and readable. "User-agent" names which bot a rule applies to, with an asterisk meaning all of them. "Disallow" lists a path the bot should not crawl; "Allow" carves out an exception inside a disallowed area. "Sitemap" points crawlers to your XML sitemap so they can find your important URLs efficiently. A typical file addresses all bots, disallows a few genuinely private or pointless areas, and links the sitemap. The rules are matched by path prefix, so "Disallow: /admin" blocks anything starting with /admin — which is powerful, and precisely why a careless slash can block far more than you intended.
What to block — and what never to
Sensible things to disallow are areas that offer no search value or that you don't want crawled at all: admin and login pages, internal search-results pages, cart and checkout flows, thank-you and duplicate utility pages. What you must never block is anything you want to rank, and — critically — the CSS and JavaScript your pages need to render. Modern crawlers render pages the way a browser does to judge them; block your stylesheets and scripts and Google may see a broken, unstyled page and rank it accordingly. Many well-meaning site owners have tanked their own visibility by disallowing resource folders they assumed were irrelevant.
The catastrophic mistake
There is one line that can erase a site from search overnight: "Disallow: /" under "User-agent: *", which tells every bot to crawl nothing at all. It's shockingly common as an accident — often left over from a staging environment where blocking everything was correct, then pushed live unchanged. Because robots.txt is invisible unless you go looking, the damage can run for weeks before anyone notices traffic bleeding away. The lesson is to treat this file with respect, always know exactly what your live robots.txt says, and check it after any site migration or launch. A single wrong character here outweighs almost any on-page optimisation you could make.
Sitemaps and crawl efficiency
Beyond blocking, robots.txt has a positive job: pointing crawlers to your sitemap. Including a "Sitemap:" line with the full URL of your XML sitemap helps search engines discover all your important pages quickly, which matters especially for large or newly-launched sites. Related to this is crawl budget — the finite attention a crawler gives your site. On a big site, stopping bots from wasting that budget on endless filtered, sorted or paginated duplicate URLs frees them to spend it on the pages that actually matter. For a small site crawl budget is rarely a real constraint, so don't over-engineer it; for a large one, thoughtful disallow rules genuinely help.
It's a public file — never use it to hide secrets
Anyone can read your robots.txt simply by visiting yoursite.com/robots.txt, including the list of paths you've disallowed. That makes it exactly the wrong place to "hide" sensitive directories: you'd be publishing a signpost that says "the private stuff is over here, please don't look," which curious people and bad actors happily ignore. Genuine protection comes from authentication, access controls and server configuration, not from politely asking crawlers to stay away. Use robots.txt to manage crawling of public pages efficiently — and never as a security measure, because it isn't one.
Always test the live file after you deploy it
Never assume your robots.txt works because it looked right when you wrote it. After it's live, fetch yoursite.com/robots.txt in a browser to confirm the deployed version matches what you intended — plenty of sites serve a stale or default file without realising. Google Search Console reports on how Google reads your robots.txt and flags URLs it was blocked from crawling, which is how you catch an accidental disallow before it drains your traffic. Make checking the live file a fixed step after every launch, migration or theme change, because this is precisely the kind of setting that gets clobbered silently and noticed only when the rankings fall.
One file by hand vs technical SEO at scale — where Rank Math does more
Writing a robots.txt once, for a straightforward site, is entirely doable — and this generator makes the syntax safe. Managing crawling as part of a living WordPress site is a broader task: editing robots.txt safely from the dashboard, generating and submitting XML sitemaps automatically, handling noindex directives on the right templates, and keeping the whole technical-SEO picture coherent as you add pages and plugins. That's where a plugin like Rank Math does more — it wires robots.txt, sitemaps and indexing controls together so the pieces stay consistent instead of drifting apart. Use this tool to produce a correct file now; use a full SEO plugin to keep crawling and indexing under control as the site grows.
Frequently asked questions
Where do I put the robots.txt file?
In your site's root, so it lives at yoursite.com/robots.txt. Upload it there or use your CMS's robots.txt setting.
Does robots.txt stop a page being indexed?
No — it stops crawling, but a blocked page can still be indexed if other sites link to it. Use a noindex tag or password protection to truly keep a page out of search.
Should I block anything by default?
Usually only admin, cart, search-result and duplicate URLs. Don't block CSS/JS or pages you want ranked. An SEO plugin like Rank Math manages this safely.
More free content & seo
Embed this tool on your site
Blogger, teacher or toolmaker? Put this calculator on your own page — free forever, no strings. Copy the snippet below (the credit link is appreciated and keeps the tool free):
This tool is free and runs entirely in your browser. The link above is an affiliate link: we may earn a commission if you sign up, at no extra cost to you, and it never changes our honest take.