faulty robots.txt???

hi guys,

my robots.txt looks like:


when I look at the logfile I see strange things: the google bot is regularly visiting and indexing sites but does only come to my site and reads the robots.txt allthe time and then leaves again. Today it read it 7 times and nothing else.

Btw to which chmode do I have to set the robots.txt?
And how do some ppl manage to get a 404 error, when retrieving my robots.txt???

Am I doing something wrong?

And thats all that slurp does on my site.

I just got an answer from inktomi tech support:



Here is my answer to them:


Maybe thsi helps others too and maybe someone else can shed some more light onto this.

talking about 404 errors:

66.77.73.162 - FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)

Date Page Status Referer
01/03 15:01 /robots.txt 404 -

The first time FAST visited me for a long time and it did not find my robots.txt ???

Of course I have one and it is ok..

I just thought this would fit in. Here is another excerpt from my access logfile:



Never had anything like that on my server!

At least googlebot and fast are now busy reindexing my site although slurp is still only interested in my robots.txt

I have beend thinking about removing the robots.txt for a week or so and see what happens. Btw this is not a critical site it is just a personal site I made to show pictures from my last travels and I am not finished with the design. The whole thing is that I am trying out my SEO skills on this unimportant page. I am trying to push my page up in the rankings just to see if I can do it. Doesn't matter if I remove robots.txt and all the crap in each and every subdirectory gets indexed. If I put the file back after one week, robots will return and if no one is linking to directories I forbid for spiders those will disappear from search engines?

I removed robots.txt for 3 days now, everythings normal except that slurp is busy with my site. it is checking all the pages it had indexd months/years ago. I hope when it is finished with those inexistent pages it will start indexing the new pages.

The conclusion: slurp had problems with my robots.txt - have a look at the first post here and see if there is something wrong with my robots.txt file.

I put robots.txt back and slurp/cat left.

now only slurp/si is visiting.



and slurp/cat which was indexing my site left. there is nothing wrong with my robots.txt and I have other content than what I excluded. Do you think I should write to them again???