Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsWeb DesignSearch Engine Optimization

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old July 9th, 2004, 10:22 AM
computer's Avatar
computer computer is offline
echo $usertitle['computer'];
Dev Shed God 4th Plane (6500 - 6999 posts)
 
Join Date: Jan 2003
Location: UK
Posts: 6,673 computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level) 
Time spent in forums: 4 Weeks 9 h 2 m 19 sec
Reputation Power: 219
Send a message via ICQ to computer
small seo question

hey.

I have an active website, however googlebot seems to be around permenantly. I often have 10-15 googlebots around at once recrawling my site. Now, I'm not too sure why they come back everyday, but they're sucking up my bandwidth like crazy horses! So far, in the 8 days of this month since I last checked the logs they've used about 300Mb of bandwidth (which doesn't sound too much, but the actual users have only used about 50Mb).

What's the easiest way to ask them politely to just visit say, once every 3 days maximum? I'm using php scripts btw with dynamic content, but is there a meta tag to do this (I think there is)... Is this a good idea?

Thanks...
__________________

Last edited by computer : July 9th, 2004 at 12:57 PM.

Reply With Quote
  #2  
Old July 9th, 2004, 11:03 AM
dave2k's Avatar
dave2k dave2k is offline
Banned
Dev Shed Novice (500 - 999 posts)
 
Join Date: May 2003
Location: GB
Posts: 553 dave2k User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 27 sec
Reputation Power: 0
http://www.seochat.com/

maybe it's someone using a fake referrer url? Do you have content that people might want to grab?

Reply With Quote
  #3  
Old July 9th, 2004, 11:07 AM
codergeek42's Avatar
codergeek42 codergeek42 is offline
少しな日本語とスペイン語と英語を話します。
Dev Shed God 2nd Plane (6000 - 6499 posts)
 
Join Date: Jul 2003
Location: Anaheim, CA (USA)
Posts: 6,415 codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)codergeek42 User rank is General (90000 - 100000 Reputation Level)  Folding Points: 35702 Folding Title: Starter FolderFolding Points: 35702 Folding Title: Starter Folder
Time spent in forums: 1 Month 1 Week 6 Days 2 h 29 m 21 sec
Reputation Power: 968
Send a message via ICQ to codergeek42 Send a message via AIM to codergeek42 Send a message via Yahoo to codergeek42 Send a message via Google Talk to codergeek42
Post

Moved.
__________________
~~ Peter ~~
( My Blog: It's exactly like normal nerdiness, but completely different. ) :: ( Supporter of the EFF & FSF ) :: ( I'm a GNU/Linux addict and Free Software Advocate. ) :: ( How to Ask Questions the Smart Way ) :: ( The Fedora Project, sponsored by Red Hat ) :: ( GNOME: The Free Software Desktop Project ) :: ( GnuPG Public Key )

Reply With Quote
  #4  
Old July 9th, 2004, 12:17 PM
computer's Avatar
computer computer is offline
echo $usertitle['computer'];
Dev Shed God 4th Plane (6500 - 6999 posts)
 
Join Date: Jan 2003
Location: UK
Posts: 6,673 computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level)computer User rank is Captain (20000 - 30000 Reputation Level) 
Time spent in forums: 4 Weeks 9 h 2 m 19 sec
Reputation Power: 219
Send a message via ICQ to computer
no not really, they're just forums...

I'm getting lots of different crawlerx.google.com I think, resolving to ips belonging to google... I'm just always under attach...

isn't there a robots revisit meta tag? will that work in this instance?

Reply With Quote
  #5  
Old July 9th, 2004, 03:09 PM
tony84 tony84 is offline
tony
Dev Shed Novice (500 - 999 posts)
 
Join Date: Apr 2004
Location: manchester uk
Posts: 670 tony84 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 1 Day 13 h 11 m 38 sec
Reputation Power: 5
Send a message via AIM to tony84 Send a message via MSN to tony84
<meta name="revisit-after" content="10 days" /> is the 1 your talking about - change the days to whatever

<meta name="robots" content="INDEX, FOLLOW" /> this might be useful read up about this as it can be used to stop the spiders going through every page of your site
__________________
Free Forum hosting for clans

Reply With Quote
  #6  
Old July 17th, 2004, 01:46 PM
dave2k's Avatar
dave2k dave2k is offline
Banned
Dev Shed Novice (500 - 999 posts)
 
Join Date: May 2003
Location: GB
Posts: 553 dave2k User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 27 sec
Reputation Power: 0
but not all engines stick to these rules - I think i saw the tags that google recognises somewhere on their site

Reply With Quote
  #7  
Old July 22nd, 2004, 01:04 AM
happyallday happyallday is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jul 2004
Posts: 129 happyallday User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 4
if you don't want your forum get indexed, just use robots.txt, otherwise, you should think about invest more bandwidth

Reply With Quote
Reply

Viewing: Dev Shed ForumsWeb DesignSearch Engine Optimization > small seo question


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway