|
|
|||||||||
|
|||||||||
| |||||||||
|
|
|
| |||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
#1
|
||||
|
||||
|
small seo question
hey.
I have an active website, however googlebot seems to be around permenantly. I often have 10-15 googlebots around at once recrawling my site. Now, I'm not too sure why they come back everyday, but they're sucking up my bandwidth like crazy horses! So far, in the 8 days of this month since I last checked the logs they've used about 300Mb of bandwidth (which doesn't sound too much, but the actual users have only used about 50Mb). What's the easiest way to ask them politely to just visit say, once every 3 days maximum? I'm using php scripts btw with dynamic content, but is there a meta tag to do this (I think there is)... Is this a good idea? Thanks... Last edited by computer : July 9th, 2004 at 12:57 PM. |
|
#2
|
||||
|
||||
|
http://www.seochat.com/
maybe it's someone using a fake referrer url? Do you have content that people might want to grab? |
|
#3
|
||||
|
||||
|
Moved.
__________________
~~ Peter ~~ ( My Blog: It's exactly like normal nerdiness, but completely different. ) :: ( Supporter of the EFF & FSF ) :: ( I'm a GNU/Linux addict and Free Software Advocate. ) :: ( How to Ask Questions the Smart Way ) :: ( The Fedora Project, sponsored by Red Hat ) :: ( GNOME: The Free Software Desktop Project ) :: ( GnuPG Public Key ) |
|
#4
|
||||
|
||||
|
no not really, they're just forums...
I'm getting lots of different crawlerx.google.com I think, resolving to ips belonging to google... I'm just always under attach... isn't there a robots revisit meta tag? will that work in this instance? |
|
#5
|
|||
|
|||
|
<meta name="revisit-after" content="10 days" /> is the 1 your talking about - change the days to whatever
<meta name="robots" content="INDEX, FOLLOW" /> this might be useful read up about this as it can be used to stop the spiders going through every page of your site
__________________
Free Forum hosting for clans |
|
#6
|
||||
|
||||
|
but not all engines stick to these rules - I think i saw the tags that google recognises somewhere on their site
|
|
#7
|
|||
|
|||
|
if you don't want your forum get indexed, just use robots.txt, otherwise, you should think about invest more bandwidth
|
![]() |
| Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > small seo question |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|