robots.txt - what if I don't want the WORLD knowing of directory?

Hi guys,

Not so much an optimization question, but to do with robots.txt.

I've written a backend administration area for a site I've developed - let's say the directory is /admin.BLAH

The area is protected using .htaccess & .htpasswd files.

Do I need to include /admin.BLAH as disallow in my robots.txt to keep bots out, all will the ht... block them anyway so there is no need?

The reason I ask is, I don't want to list the area in my robots.txt as that hands the location over to would be hackers. If I MUST list it in robots.txt, is there any wildcard that might work as well?

I read somewhere that wildcards do work (i.e. /bob will block /bob.html & /bob/blah.html). If I added simply /ad - would that disallow /admin.BLAH as well?

Thanks for any help you can provide!

I would have to say no.

Since the directory is .htaccess protected the robot, even if it found the admin.blah directory cannot see anything inside of it.

The web server will only return information(web content) IF it recieves a valid login. Otherwise the it returns a 403 Error. Which most robots would just ignore anyway


Hope that helps

Cheers dude, much appreciated. Thought that would be the case.