Robots.txt and sub-domains

Hi All,

I have a number of websites stored in sub-domains of the main domain (ie /htdocs/<sub-domain name>).

My understanding is that the robots.txt file must appear in the main (/htdocs) folder and that the file contains the URL of the website.

Questions:
* Am I right in saying that the URL should appear in the robots.txt file?
* If I am, how do I deal with the sub-domains?
* Does every sub-domain need its own robots.txt file and, if so, how do I get the search engines to look somewhere other than the /htdocs folder?

Thanks for any help.

Regards,

Hywel

Each subdomain needs to have its own robots.txt if it is going to be accessed as a subdomain, ie sub.domain.com

If you're accessing it using www.domain.com/sub then it doesn't need one and can use the base domain's robots.txt

But the whole point of a subdomain is to be used as a sub, so yes. It needs its own.

More information about robots.txt

I actually gave that robogen program a try and it was pretty interesting.

http://forums.devshed.com/robots.txt
http://devshed.com/robots.txt (blank robots.txt)