
October 22nd, 2005, 08:50 AM
|
|
Registered User
|
|
Join Date: Jul 2004
Posts: 8
Time spent in forums: 2 h 36 m
Reputation Power: 0
|
|
|
FAST Enterprise Crawler 6 - what's it up to with our site??
Hi,
Started up a new site a while back and the bigger crawlers like google, msn, yahoo, ask etc. are coming back pretty much daily for their doseful of spidering. All obviously more than welcome.
However, on 20th October, very early in the morning we started to get spidered by the following robot: FAST Enterprise Crawler 6 used by fast (morten at fast dot no) . The bot entered via the homepage and then progressed to visit a page every minute or so for about 24 hours. So about 1,400 pages of my 2,000 page site.
It stopped, as I said, after 24 hours. Just over 24 hours after that though (i.e. morning of 22nd), it returned and went through some more URLs at the rate of a page a minute and is continuing to do so as I type.
Now, this isn't overloading me at the moment at all, so I didn't panic and exclude or restrict via robots.txt once I saw the logs. However, I am *very* curious. From my half-bit knowledge, I seem to remember that FAST was for alltheweb, until it then switched to Yahoo's crawl data. So, what is this user agent and is it worth allowing access?? Visits to the fast.no website is unhelpful, at least for a novice like me.
What's going on?
|