checking bots

spidering me website?
but still not indexed yet in google.

check my website on google.
www.travelwalk.net

Saw these bots in my awstats. one of search engines' spiders?
need to verify if they are genuine frm search engines with u guys...

-Inktomi Slurp

-Alexa (IA Archiver)


ps. guys, thanks for the help so far. everything's starting to make sense thes days.

relax and wait. Google takes some times to process the pages it crawled.

Type in your domain name on google... it's already indexed on google. Nice, you have a lot of pages indexed there. You should exchange links to raise your PR.

dejaone is right, when i first submitted our site web2txt.co.uk to Google there were a few weeks of anticipation till we could see ourselves listed. it's frustrating that with the speed of the net it takes soo long to get a result.

Google came, then visitors and now we're working out how to get more traffic from places other than Google. There have been times when it's been like having all our eggs in one basket, every so often Google drops em and heck do we know about it.

Here's a handy list of robots:

googlebot
scooter
henry
slurp
fast
infoseek
ask
tecnoseek
crawler
firefly
rabaz
zyborg
spade
webbug
appie
galaxy
girafabot
webfindbot
nationaldirectory
teoma
gulliver

I just happened to have them on my desktop, hope they come in handy for someone. We work out if it's a spider using php and eregi, comes in handy to see who's spidering, how many pages and how often.

cool list of bots

and yes, my site's been indexed. sweet.
Took a while for google to index. heheh

thanks guys for the advice. I am working on my link exhange now. hope ranking goes up.!

also, I have just started working on robot.txt
not sure.

does this look ok to you guys?

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /private/

Disallow: forums/post-*.html$
Disallow: forums/updates-topic.html*$
Disallow: forums/stop-updates-topic.html*$
Disallow: forums/ptopic*.html$
Disallow: forums/ntopic*.html$

suggestions?

If in doubt check it out - there'll be stacks of sites there who will validate your robots.txt so see if it's well formed.

Hope this helps.

It usually takes a week+ (assuming the site is well linked to) for new pages to show up in Google's cache.