Inside of Webmaster Tools, the Google Robots.txt Generator has been unveiled. The Robots.txt Generator will help webmasters travail the often misunderstood method of allowing or disallowing search engine robots from crawling files, folders and pages on a website.

Inside of the tool, webmasters are offered the Action of allowing or blocking plus another pull-down menu gives the choices of All Robots, Googlebot, Google-bot Mobile, Googlebot-Image, Mediapartners-Google, Adsbot-Google or Other robot and you can input the name of this bot in the field.

There is next a field for files or directories to fill in so use the relative path. Once adding this you’ll see a the robots.txt file displayed in a window that will look like this:

User-Agent: *
Allow: /

User-Agent: Googlebot
Disallow: /directory/file.html
Allow: /

Just upload this file to the website’s root directory and you’re done. It’s pretty simple and it gives webmasters a no-guess way to allow or block the bots at will.

Since this is April 1, hopefully the useful Google Robots.txt Generator tool is not an April Fool’s joke such as the Virgle Mars trip they are promoting today.

Filed under: Google