What Is a Robots.txt File and How to Create One
A robots.txt file is a small text file that tells search engine crawlers which parts of your website they may visit. It sits at the root of your site and is one of the first things well-behaved crawlers look for. This guide explains what it does, how the rules work, and how to create one safely.
What robots.txt does
Robots.txt controls crawling, which means whether a bot is allowed to request a page. It is useful for keeping crawlers out of areas that have no value in search, such as admin pages, internal search results or duplicate parameter pages, and it helps you avoid wasting crawl time on them. It also lets you point crawlers to your sitemap.
What it does not do
Robots.txt is not a security tool and it is not a reliable way to keep a page out of search results. The file is public, so anyone can read it, and a blocked URL can still appear in results if other sites link to it. To keep a page out of the index, use a noindex meta tag or password protection instead, and make sure the page is not blocked, because crawlers must be able to see the tag.
Where the file goes
The file must be named robots.txt in lowercase and placed in the root of your domain, for example https://example.com/robots.txt. A file placed in a subfolder is ignored.
The main rules
- User-agent names the crawler the rules apply to. An asterisk means all crawlers.
- Disallow lists a path the crawler should not visit.
- Allow makes an exception inside a disallowed folder.
- Sitemap gives the full address of your XML sitemap.
Common examples
Allow everything:
User-agent: * Disallow: Sitemap: https://example.com/sitemap.xml
Block one folder:
User-agent: * Disallow: /admin/ Sitemap: https://example.com/sitemap.xml
Block everything (use only on a staging site):
User-agent: * Disallow: /
Mistakes to avoid
- Leaving "Disallow: /" on a live site. This blocks your entire website and is a common launch mistake.
- Blocking CSS or JavaScript files. Search engines need them to render your pages properly.
- Using robots.txt to hide private data. Anything listed in it is visible to everyone.
- Typos in paths. Paths are case sensitive, so check them carefully.
How to create and test yours
You can write the file by hand in a plain text editor, or use our Robots.txt Generator to build one with a few clicks. Upload it to the root of your site, then open yourdomain.com/robots.txt in a browser to confirm it loads. After that, use the robots.txt report in Google Search Console to check for errors. New to the bigger picture? Read What Is SEO and How Does It Work.
Conclusion
A good robots.txt is short and simple: allow what should be crawled, block only what truly should not be, and link to your sitemap. Check it after every site change, because one wrong line can hide your whole site.