vBulletin robots.txt
The robots.txt file is something that almost all web sites should have. It is a file that tells the search engine crawlers what pages on a web site that they should crawl or should not crawl. This conserves bandwidth in having them not crawl web pages that are of no use to the search engines. It also help prevent the search engines index two copies of the same think. For example, the original version of a page and the print version of a page. Some software, like vbulletin forum software have a lot of pages that do not need to be indexed, such as the registration page. The vBulletin robots.txt is a valuable addition to these sites.
