robots.txt

Sorry if this is in the wrong forum but,

I always go to my error log, and robots.txt always comes up. I am sure this has rto do with the google bots, and they go there for certain informatiuon or something, and I really do not want this error to keep popping up, I always want my error page spotless. So what do I do with this robots.txt and how do I use it?

Pretty much just explain it to me, thanks in advance.

All replies are appreciated.

You may find that the robots.txt file will be helpful for your site(s)

http://www.searchengineworld.com/ro...ts_tutorial.htm

Moving to General Web Design

Thanks for the replies, what do you think I should add to the robots.txt. I am currently thinking of just adding:

User-agent: *
Disallow:

But I am new to this and I am unaware of what to add.

One of the purposes of the file is to direct the crawler to certain
directories for search engines.

If you used Disallow * then you would not be listed in any search
engines.

I use it like the following:

User-agent: googlebot
Disallow: /banners/
Disallow: /downloads/
Disallow: /include/
Disallow: /images/
User-agent: *
Disallow: /banners/
Disallow: /downloads/
Disallow: /include/
Disallow: /images/

This is telling the crawlers to stay out of the listed folders, not to
index them in a search.

My understanding is that you might want a different listing for the googlebot (User-agent: googlebot)
and then one for all the rest (User-agent: *)

The tutorial link above should be more than enough info for you. Remember
that the file goes into your root web directory:
domain.com/robots.txt