Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed Forums > Web Design > Search Engine Optimization
Reload this Page

A Question About Robots.

Discuss A Question About Robots. in the Search Engine Optimization forum on Dev Shed.
A Question About Robots. Search Engine Optimization forum discussing related topics such as Google, PageRank, Link Exchange, and Link Popularity. Improve your ranking in the Search Engines by employing good optimization techniques.

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
« Previous Thread | Next Thread »  
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old February 12th, 2007, 04:41 AM
g_hadgraft g_hadgraft is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Mar 2005
Posts: 172 g_hadgraft User rank is Private First Class (20 - 50 Reputation Level)g_hadgraft User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 1 Day 12 h 1 m 21 sec
Reputation Power: 4
A Question About Robots.
I have a quick question about robots.

My question is this:

When robots crawl a site will they check the robots.txt once when they enter the site, or will they check the robots.txt for each page they crawl?

I cant seem to find an answer on this.

Thanks, in advance.
Reply With Quote
  #2  
Old February 12th, 2007, 09:49 AM
stymiee's Avatar
stymiee stymiee is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2003
Posts: 225 stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level)stymiee User rank is Sergeant Major (2000 - 5000 Reputation Level) 
Time spent in forums: 20 h 48 m 23 sec
Reputation Power: 49
They check it once and then they only crawl the pages not blocked within it.
Reply With Quote
  #3  
Old February 23rd, 2007, 11:12 AM
etgsgroup etgsgroup is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Feb 2006
Posts: 2 etgsgroup User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 27 m 50 sec
Reputation Power: 0
but some bad site robot not read robots.txt .
Good search robot will read robots.txt first.
Reply With Quote
  #4  
Old February 23rd, 2007, 11:20 AM
jbjacob's Avatar
jbjacob jbjacob is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Oct 2006
Location: United Kingdom
Posts: 172 jbjacob User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 Days 15 h 26 m 37 sec
Reputation Power: 3
Send a message via ICQ to jbjacob Send a message via MSN to jbjacob
Every ethical robot first reads the robots.txt and then crawls the webpages.
__________________

VPS Hosting | Dedicated Server Hosting
Email & MSN :: ryanw @ eUKhost.com || AIM :: RyanUKweb
Complete Web Hosting Solutions at eUKhost.com
Reply With Quote
  #5  
Old March 11th, 2007, 09:55 PM
McMEnterpr McMEnterpr is offline
Registered User
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Nov 2006
Posts: 13 McMEnterpr User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 5 h 54 sec
Reputation Power: 0
Some bad robots won't read your txt and will eat a lot of bandwidth looking through your site. If that's a problem, you can deny these bad robots access.
Reply With Quote
  #6  
Old March 13th, 2007, 01:36 AM
WebGeek182's Avatar
WebGeek182 WebGeek182 is offline
I Know Kung Fu
Dev Shed Newbie (0 - 499 posts)
  
Join Date: Feb 2007
Location: In the Matrix
Posts: 38 WebGeek182 User rank is Private First Class (20 - 50 Reputation Level)WebGeek182 User rank is Private First Class (20 - 50 Reputation Level) 
Time spent in forums: 8 h 16 m 27 sec
Reputation Power: 2
Some bad robots will even read your robots.txt looking for what's off limits and head straight there.
Last edited by WebGeek182 : March 13th, 2007 at 02:39 AM.
Reply With Quote
Reply

Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > A Question About Robots.

« Previous Thread | Next Thread »

Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Linear Mode Linear Mode
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump

 Free IT White Papers!
 
How to Present Effectively Online
This white paper offers practical and actionable advice on the key steps that any presenter should consider as they plan and execute a Webinar or online meeting.
 
Open Source Security Myths
Open Source Software (OSS) is computer software whose source code is available to the general public with relaxed or non-existent intellectual property restrictions (or arrangement such as the public domain), and is usually developed with the input of many contributors.
 
Power and Cooling Capacity Management for Data Centers
This paper describes the principles for achieving power and cooling capacity management.
 
Scalable, Fault-Tolerant NAS for Oracle - The Next Generation
For several years NAS has been evolving as a storage alternative for Oracle databases, and for good reason: NAS is quite often the simplest, most cost-effective storage approach for Oracle. Learn about the benefits that HP's approach to scalable NAS brings to Oracle environments in this comprehensive white paper.
 
Understanding Web Application Security Challenges
This white paper discusses many common threats and preventive measures for Web application security, and explains what you can do to help protect your organization.
 

Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 3 hosted by Hostway. 
Stay green...Green IT