Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Google Crawling Forums

Discuss Google Crawling Forums in the Search Engine Optimization forum on Dev Shed.
Google Crawling Forums Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old September 6th, 2004, 11:55 PM
falcon1 falcon1 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jul 2003
Posts: 139 falcon1 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 12 h 37 m 2 sec
Reputation Power: 6
Google Crawling Forums
Hey everyone! In the past couple of days I have noticed that there is always the same guest hanging around on my forum, so I decided to find out a little about them (mainly where they are coming from). So I typed in their IP address into an IP lookup site, and apparently it was the google spider.

It seems that it is continually crawling through my forums. I would like to know if this could cause the pages to load slower for other people (because of the added requests to my server). If so, is there a way to stop google from crawling just the forums? Thanks for any advice!

-Jonathan
Reply With Quote
  #2  
Old September 8th, 2004, 10:18 AM
tw1sta tw1sta is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2004
Posts: 3 tw1sta User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
Quote:
Originally Posted by falcon1
Hey everyone! In the past couple of days I have noticed that there is always the same guest hanging around on my forum, so I decided to find out a little about them (mainly where they are coming from). So I typed in their IP address into an IP lookup site, and apparently it was the google spider.

It seems that it is continually crawling through my forums. I would like to know if this could cause the pages to load slower for other people (because of the added requests to my server). If so, is there a way to stop google from crawling just the forums? Thanks for any advice!

-Jonathan


block the ip from the .htacces file or from (if) cpanel control panel.
Reply With Quote
  #3  
Old September 9th, 2004, 07:30 AM
tidemann tidemann is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Sep 2004
Location: Oslo, Norway
Posts: 24 tidemann User rank is Private First Class (20 - 50 Reputation Level)tidemann User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 36 m 32 sec
Reputation Power: 0
You can also block the any search engine spider in the robots.txt file.
Reply With Quote
  #4  
Old September 10th, 2004, 12:54 PM
falcon1 falcon1 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jul 2003
Posts: 139 falcon1 User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 12 h 37 m 2 sec
Reputation Power: 6
Thanks guys!! Will give those suggestions a go.
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Google Crawling Forums

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway.