|
|||||||||
|
|||||||||
| |||||||||
|
|
|
| |||||||||
 |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
Dev Shed Forums Sponsor:
|
|
Get inside! Sample the range of functionality easily built with JMSL Library for Time Series Data Analysis, Heat Maps, Portfolio Optimization, Monte Carlo Simulation, Stock Price Charting and more. Download Now! |
|
#1
April 15th, 2008, 07:09 PM
|
|||
|
|||
Webcrawlers, a big problem
I need some serious help. My website is being crawled 24x7 and that reduces the load time of the website to a point, where 30% of the traffic is showing as error code 206 in my awstats. I need to stop these automated crawlers from accessing my site.
These crawlers may include email extractor, data miners etc. Any help will be highly appreciated.
|
|
| Viewing: Dev Shed Forums > Operating Systems > UNIX Help > Webcrawlers, a big problem |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
 |
|
|
|
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
Linear Mode
