CueBlocks Robots.txt extension allows you to edit the robots.txt file right from your admin interface. Robots.txt is the de-facto web-wide standard to control crawling and indexing of web-based content. All instructions to web robots need to be placed in this file which is placed in the root folder of your website. Remember that robots can choose to follow or ignore the instructions. You are also required to provide these instructions in a specific format. For more information, please go to http://en.wikipedia.org/wiki/Robots.txt
You can directly download this extension from magento-connect.