Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

Spiders...

Discuss Spiders... in the Search Engine Optimization forum on Dev Shed.
Spiders... Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old April 24th, 2006, 09:46 PM
mrsbuzzy mrsbuzzy is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Apr 2006
Posts: 15 mrsbuzzy User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 6 h 58 m 39 sec
Reputation Power: 0
Question Spiders...
Do any of you have feelings on Search Engine Spiders (particularily for forum sites). We are moving to our own server. Before that we had to turn them off because of excess server pull.
Reply With Quote
  #2  
Old April 25th, 2006, 01:09 AM
Noodles23's Avatar
Noodles23 Noodles23 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Jun 2004
Location: Auckland, NZ
Posts: 426 Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)Noodles23 User rank is Major (30000 - 40000 Reputation Level)  Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2Folding Points: 886093 Folding Title: Super Ultimate Folder - Level 2
Time spent in forums: 2 Days 23 h 23 m 29 sec
Reputation Power: 387
search engine spiders are great, without them none of our sites would be in any search engines and no one would be able to find them!

If a certain spider is giving you grief put some instructions in robots.txt for it

http://www.robotstxt.org/wc/robots.html
__________________
Cooking Forums
Reply With Quote
  #3  
Old April 25th, 2006, 07:19 PM
Gnome101's Avatar
Gnome101 Gnome101 is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Dec 2004
Posts: 316 Gnome101 User rank is Corporal (100 - 500 Reputation Level)Gnome101 User rank is Corporal (100 - 500 Reputation Level)Gnome101 User rank is Corporal (100 - 500 Reputation Level)Gnome101 User rank is Corporal (100 - 500 Reputation Level) 
Time spent in forums: 3 Days 12 h 24 m 11 sec
Reputation Power: 5
If you are worried about server pull, have them not have access to you pics folder, and/or your administrative back-ends (if you have any in your wwwroot).
Reply With Quote
  #4  
Old April 27th, 2006, 12:47 PM
dotancohen's Avatar
dotancohen dotancohen is offline
המבין יבין
Dev Shed Regular (2000 - 2499 posts)
  
Join Date: Jul 2001
Location: Haifa
Posts: 2,032 dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level)dotancohen User rank is General 5th Grade (Above 100000 Reputation Level) 
Time spent in forums: 2 Weeks 2 Days 1 h 53 m 49 sec
Reputation Power: 1301
Send a message via ICQ to dotancohen Send a message via AIM to dotancohen Send a message via MSN to dotancohen Send a message via Yahoo to dotancohen
If you have a forum site then make sure that the spiders can't post! I've had it happen.

You will not want to block to many spiders becuase without them the search engines won't know that you exist. Which specific spiders were causing you trouble?
__________________
. . . What is Firefox?
. . . . . . What is Linux?
. . . . . . . . . . . What is Love?
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > Spiders...

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 1 hosted by Hostway. 
Stay green...Green IT