Google Brings Robot.txt generator to the Webmaster Tools

Google Webmaster Tools has a new tool that would certainly be useful to site
owners in making their site Googlebot friendly – the Robots.txt generator.
Basically, what this translator does is to automatically generate robot.txt
file. With a few clicks of the mouse, you can easily have your site’s
robots.txt file. Thus it eliminates the technicalities of creating Robots.txt
that has haunted novice webmasters like me before.

Using the Robots.txt
generator, webmasters can easily instructs any robots which files or
folders in your site’s root directory should be crawled by the Googlebot. You
can even choose which specific robot you want to have access to your site’s
index and restrict other robots from doing the same thing. Similarly, you can
further refine this crawling activities by specifying which robot should access
certain files in your root directory and which robot should access another
file.

Although this might seem a pretty useful tool, there are still some
limitations to its application. Some robots may ignore instruction in the
robots.txt and continue to craw your site including those files or folders that
have restrictions. So for highly sensitive files or documents, it would still
be wise to put them behind password protection.
Although it is guaranteed that Googlebots will recognize your robots.txt based
on the guidelines that you indicated when you were generating your robots.txt,
other major robots may actually ignore it.

But still, the robots.txt generator is a great
addition to the Webmaster Tools, as it will definitely make the lives of
webmasters a little bit easier than before