Robots.txt Generator
Create crawl rules and save the result as robots.txt in your site's root folder.
How to use the Robots.txt Generator
Choose whether crawlers may access your site, list private folders to block, add your sitemap address, then download the file. Upload robots.txt to your site root so it opens at yourdomain.com/robots.txt. Never block pages you want to appear in search.
Frequently asked questions
Where do I put robots.txt?
In your site root, so it opens at yourdomain.com/robots.txt.
Does robots.txt hide pages from Google?
Not reliably. It asks crawlers not to visit pages, but a blocked page can still appear if other sites link to it. Use a noindex tag to keep a page out of results.
Should I add my sitemap?
Yes. Adding a Sitemap line helps crawlers find all your pages.
Read the guide
What Is a Robots.txt File and How to Create One
More free tools
Meta Tag Generator · Word & Character Counter · Slug Generator · Keyword Density Checker · Text Case Converter · Password Generator · Lorem Ipsum Generator