|
|||||||||
|
|||||||||
| |||||||||
|
|||||||||
|
|||||||||
| |||||||||
|
|
|
| |||||||||
 |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
Dev Shed Forums Sponsor:
|
|
|
|
#1
August 9th, 2006, 02:40 PM
|
|||
|
|||
Ideal throttling values not to lockout Googlebot
The CMS I use has a throttling feature with 2 settings :
Maximum Number of Page Loads The total number of times a user is allowed to load any of your web pages (within the time interval below) before being locked out. Time Interval (in seconds) The number of seconds during which the above number of page loads are allowed. This is great to prevent DOS attacks but it can also prevent Googlebot from indexing more than a couple of pages, and that's no good. What safe values do you advise ? If you are not sure, please don't advise me anything.
|
|
#2
August 9th, 2006, 09:04 PM
|
||||
|
||||
|
You'll have to talk to the people who made the CMS to see if there's an exception for search bots. It's kind of a bad way to do DoS protection as it does nothing to protect against things like syn attacks, just stops people from spidering your site/using up all the resources that way.
__________________
Cooking Forums
|
|
#4
August 9th, 2006, 10:09 PM
|
||||
|
||||
|
I'd suggest to the vendors that they include one, but for now try throttling google bot/other web bots with robots.txt so they don't exceed the limit set by your CMS
Code:
User-agent: * Crawl-delay: 10 Taken from http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html, I'm unsure if it'll work for googlebot, best to check google's site
|
|
| Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Ideal throttling values not to lockout Googlebot |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
 |
|
|
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
Linear Mode
