What Is a Robots.txt File and How to Create One
When a search engine visits your website, one of the first things it looks for is a small text file called robots.txt. This file gives crawlers basic instructions about which parts of your site they may visit. It is simple, but a single wrong line can hide your whole site from search engines. This guide explains what robots.txt is, how to write it, and how to avoid the mistakes that cause real problems.
What is a robots.txt file?
Robots.txt is a plain text file stored in the root folder of your website. It follows a standard called the Robots Exclusion Protocol. Crawlers, also called bots or spiders, read it before exploring your pages. The rules inside tell them where they are welcome and where they should not go. Well-behaved crawlers such as Googlebot and Bingbot follow these rules, although the file cannot force anyone to obey.
Where it must be placed
The file has to live at the top level of your domain, so it opens at an address like yourdomain.com/robots.txt. A file in a subfolder will not be found. The name must be written in lowercase, exactly as robots.txt. If you use a hosting file manager, put it in the same folder as your homepage file.
The basic syntax
A robots.txt file is made from a few simple lines.
- User-agent names the crawler the rules apply to. An asterisk means all crawlers.
- Disallow lists a path crawlers should not visit.
- Allow lists a path that is permitted, often used inside a blocked folder.
- Sitemap gives the address of your XML sitemap.
Common examples
Allow everything. This is the default for most small websites:
User-agent: * Disallow: Sitemap: https://yourdomain.com/sitemap.xml
Block one folder. Useful for admin areas or private pages:
User-agent: * Disallow: /admin/ Disallow: /private/
Block everything. Only use this on a site that is under development:
User-agent: * Disallow: /
That last example is dangerous on a live site, because it asks every crawler to stay away from all of your pages.
What robots.txt does not do
Many beginners misunderstand this file. It is not a security tool. Anyone can open it and read the paths you listed, so never use it to hide sensitive information. It also does not reliably remove pages from search results. If another website links to a blocked page, Google may still list the address without a description. To keep a page out of results, use a noindex tag on the page and allow crawlers to read that tag. Blocking a page in robots.txt can stop them from ever seeing it.
How to create a robots.txt file
- Decide which areas, if any, you want to keep crawlers out of. For a small blog, that is often nothing, or just an admin folder.
- Use our Robots.txt Generator to choose your rules and add your sitemap address.
- Download the file and keep the name exactly as robots.txt.
- Upload it to the root folder of your site.
- Open yourdomain.com/robots.txt in your browser to confirm that it shows your rules.
Mistakes that cause trouble
- Leaving "Disallow: /" in place after launching a site, which blocks everything.
- Blocking CSS and JavaScript files. Search engines need them to display your pages properly.
- Putting the file in the wrong place so crawlers never find it.
- Mixing up upper and lower case. Paths are case sensitive.
- Using it to hide private data. Use passwords instead.
How to test it
After uploading, load the file in your browser and read it carefully. Then open Google Search Console and check the robots.txt report to see whether Google fetched it without problems. If you changed an important rule, use the URL Inspection tool on a key page to confirm that it is still allowed.
Conclusion
Robots.txt is small, but it deserves care. For most websites the best file is a short one that allows crawling and points to your sitemap. Block only the folders that truly do not belong in search, never rely on it for secrecy, and always test after you change it. Our generator makes the file in a few seconds, and a quick check afterwards keeps your site safe.