robots.txt Generator

Tell search engines what to crawl. Choose your rules, add the paths to block and your sitemap, and this builds a valid robots.txt to copy straight into your site's root.

Need more than the free basics? Editing robots.txt by hand is fine once. To manage crawling, sitemaps and the rest of technical SEO across a WordPress site, an SEO plugin like Rank Math handles it from the dashboard.

Try Rank Math →

Affiliate link — we may earn a commission if you sign up, at no cost to you. It never changes our honest take.

What robots.txt does (and doesn't)

robots.txt tells crawlers which paths to skip — useful for admin, cart or duplicate URLs. But it's a crawl instruction, not a security or privacy control: blocked pages can still be indexed if linked elsewhere, and the file is public. Don't use it to hide sensitive pages (use noindex or auth for that). Always include your sitemap line so search engines find your content fast.

Frequently asked questions

Where do I put the robots.txt file?

In your site's root, so it lives at yoursite.com/robots.txt. Upload it there or use your CMS's robots.txt setting.

Does robots.txt stop a page being indexed?

No — it stops crawling, but a blocked page can still be indexed if other sites link to it. Use a noindex tag or password protection to truly keep a page out of search.

Should I block anything by default?

Usually only admin, cart, search-result and duplicate URLs. Don't block CSS/JS or pages you want ranked. An SEO plugin like Rank Math manages this safely.

This tool is free and runs entirely in your browser. The link above is an affiliate link: we may earn a commission if you sign up, at no extra cost to you, and it never changes our honest take.