Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

This robots.txt generator builds crawl rules in seconds. You add allow and disallow rules. You pick user agents like Googlebot. You add your sitemap URL at the end. The tool writes clean valid syntax for you. No coding skills are needed at all. One wrong slash can block your whole site. The generator keeps the format right. Site owners use it to guide search bots. You can block admin and login pages. You can keep staging areas out of search. You can point crawlers to your sitemap. It is free and needs no signup. Build the file in under a minute. Copy it to your site root. Update the rules any time.

How to use

1. Add a rule for each path. Start with the folders you want to block.
2. Choose allow or disallow for each rule. Disallow keeps bots out.
3. Set the user agent for the rule. Use a star for all bots.
4. Pick agents like Googlebot for specific rules. Different bots can get different rules.
5. Enter your sitemap URL if you have one. Use the full address.
6. Click the generate file button. The clean file appears at once.
7. Copy the output to your site root. Name the file robots.txt exactly.

Why use it

• You control which pages bots crawl. Keep private areas out of search.
• You block admin and login pages. They have no place in results.
• You point crawlers to your sitemap. New pages get found faster.
• You avoid syntax errors in rules. One typo can block everything.
• You create the full file in under a minute. No manual writing needed.
• You can update rules any time for free. Changes take only seconds each time.

Tips

• Test the file in Search Console after upload. Google shows how it reads your rules.
• Never disallow your whole live site by mistake. That rule belongs on staging only.
• Keep the sitemap line at the end of the file. Update it when the URL moves.
• Use specific paths not wild guesses. Check each rule before you save.

Frequently asked questions

What is a robots.txt file

It tells search bots which pages they may crawl. It sits in your site root folder as plain text. Bots read it before they crawl any page. It is a simple text file with basic rules.

Where do I upload the file

Place it at the root of your domain. It must be named robots.txt exactly as shown. Most web hosts offer a file manager for uploads. Test it in Search Console after you upload.

Can robots.txt block Google fully

A disallow all rule blocks all crawling. Use full blocks only on staging sites. Never block your live site by mistake. Double check every single rule before you save.

Should I add my sitemap here

A sitemap line helps crawlers find pages. Add one clean line with the full sitemap URL. Keep the line at the end of the file. Update it when your sitemap URL moves.

Do all bots obey robots.txt

Good bots like Googlebot obey it. Bad bots may ignore it completely. Robots.txt is a request not a lock. Use passwords for truly private pages.

Can I block one folder only

Add a disallow rule for that folder path. Other folders stay open to bots. Keep the path exact and complete. Test the result in Search Console.

What does the star user agent mean

The star means the rule applies to all bots. It is the default catch all setting. Specific agents like Googlebot override it. List specific agents above the star rule.

Build your file and place it at your site root. Test it in Search Console. Your crawl rules are now clean and clear.