A Guide to Robots.txt

This guide takes a look at robots.txt, an important but small file that will help you control search engine spider.