09/26/2026
A robots.txt file gives instructions to search engines. It tells crawlers which parts of your site they may visit. It lives at the root of your website. Every site should have one.
It does not make pages private. It only asks crawlers to stay out. True privacy needs passwords or login walls. Keep this difference clearly in mind.
Search crawlers visit your site to read its pages. Before they start they look for this file. The file lists rules for different crawlers. Each rule names an area that is off limits.
You can use it to protect thin pages. Admin areas and search result pages add no value in Google. Blocking them saves your crawl budget. Your best pages then get crawled more often.
You can also use it to guide crawlers. A final line can point to your sitemap. This helps crawlers find your page list fast. It pairs well with a good sitemap.
Crawlers request the file before crawling anything else. They read the rules that apply to them. Then they skip the blocked paths. They crawl the rest as normal.
Most big crawlers respect the rules. Google and Bing follow them closely. But the file is a request and not a lock. Bad bots can ignore it if they want.
The file is public to everyone. Anyone can type the address and read it. Never list secret paths inside it. That would only draw attention to them.
Creating the file is simple. It is plain text with a fixed name. Follow these steps to make one that works. Test it before you rely on it.
You can skip the manual work. Our free robots.txt generator builds the file for you. Answer a few simple prompts. Download the file and upload it.
Most sites need only a few basic rules. Start simple and add more only when needed. Each rule should have a clear reason. Random rules can cause harm.
If you do not have a sitemap yet make one now. Use our XML sitemap generator to build it in minutes. Then add its address to your robots file.
Test every rule after you add it. Use the tester inside Search Console. It shows which URLs a rule blocks. Fix surprises before they cost you traffic.
A wrong rule can hide your whole site. These errors happen more often than you think. Review your file after every change. One typo can cost you traffic.
Keep a backup of each version. Note the date when you change a rule. Small teams forget why a rule exists. A short log saves hours of guessing.
Whether robots.txt keeps pages out of Google
It does not fully keep them out. It stops crawling but pages can still be indexed. Google may index a blocked page if others link to it. Use a noindex tag for pages you want gone.
Where the robots.txt file must be placed
It must sit in the root folder. That means right after your domain name. Crawlers only check that one spot. A file in a subfolder will be ignored.
Whether every site needs a robots.txt file
Every site should have one. Even a simple allow rule helps. It confirms to crawlers that you thought about it. It also gives you a place to list your sitemap.
How long it takes Google to follow new rules
It can take a few days. Google refetches the file on its own schedule. You can ask for a refetch in Search Console. Big sites may wait longer.
Whether blocking a page deletes it from the index
Blocking does not delete the page. Blocking only stops the crawl. The page can stay in the index. Remove the block and add noindex instead if you want it gone.
Create your robots.txt file today and test it. Pair it with a clean sitemap for best results. Read more SEO guides to keep your site healthy.
