Test Robots.txt File Up Before Adding It To Your Website

If you have or want to hide some content on your website from Google search, you don't want the content to be indexed by google search, definitely robots.txt file is the perfect choice for you. By adding robots.txt file to your website, you'll be able not only to hide a single or some contents easily, but you'll even be able to hide as many directories as you want. After all, you will even be able to hide the whole contents on your website if you want to.

Here i'm not gonna talk about How to create and add robots.txt file onto a website, here i'm gonna talk about How to test robots.txt file so you know if the robots file you have created is realy working as expected before you add it onto your website, cause a single wrong parameter in the file could get your whole contents, your website, unindexable of google search, it's the same term with blocked by google search i think.

However, i've written a complete article due to creating and adding robots.txt file onto a website if you want to know about it, take a look on the following article: Create And Add Robots.txt File To Website

How To Check Out If A Robots.txt File Is Working As Expected

Actually the question shouldn't be how i think, but where. If you want to check out a robots.txt file so you can make sure that the file is okay for your website, you can use Google Webmaster's robots.txt tester. It's easy to use, other robots.txt testers out there are easy to use.

Go to Google Webmaster > Click on any screenshot of your site > Health > Blocked URLs

You'll find there are two text boxes here, the first one is to be filled with your parameter, your robots.txt file, and the second box is to be filled with your website address. The both boxes are normally filled outomatically, and there is a warning too on the page if your website has any trouble with robots.txt file, like Googlebot is blocked from http://j-smith-site.blogspot.com/

However, you are going to test a new robots.txt file here right. Well, you are going to fill the text boxes yourself, and you know already what are the text boxes must be filled with, don't you?

After you fill the both boxes, then click test to see the results. You can write as long parameter as you want in the first box, and you can list as many websites as you want in the second box.

If the parameter you entered into the text box prevents google search from crawling your website, or in other words it's blocking google search, you'll have a warning like the following one:

http://j-smith-site.blogspot.com/
Blocked by line 5: Disallow: /*#
Detected as a directory; specific files may have different restrictions
Blocked by line 5: Disallow: /*#
Detected as a directory; specific files may have different restrictions

All above means, that, Google search is blocked by line 5 of the parameter, in the example above it's /*#.

When you are writting, and you jump one paragraph down, the empty space in betwen two paragraphs, in robots.txt file it means one line. There is no term paragraph in robots.txt file, it's lines.

The following, what you'll have if the parameter is good.

http://j-smith-site.blogspot.com/
Allowed
Detected as a directory; specific files may have different restrictions
Allowed
Detected as a directory; specific files may have different restrictions

There are two User agents (say it web crawlers) provided for the test here, Googlebot and Googlebot-mobile, that's why the test result is double.

Well done! Now you know everything about robots.txt file, you can create one, add it to your website, or just how to test it. If you still face any trouble with robots.txt file, you tell me by throwing a comment below, i can help you get the trouble right.