Search Engine Optimization
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
User Name:
Password:
Remember me
Go Back   Dev Shed ForumsWeb DesignSearch Engine Optimization

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Rate Thread Display Modes
 
Unread Dev Shed Forums Sponsor:
  #1  
Old January 17th, 2005, 12:37 PM
frankieluccini's Avatar
frankieluccini frankieluccini is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jan 2004
Posts: 374 frankieluccini User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 Days 9 h 4 m 53 sec
Reputation Power: 5
robots_txt vs robots meta

is there any advantage in using a robots.txt file opposed
to just including a robots meta tag ?

it seems that there may be, if you want the spider to
view specific sub-directories and avoid others, but im only
trying to focus on the index page of my site.
...... index, nofollow .......

can anyone provide any input for me?
should i be ok with only including the meta, or are the spiders
more perceptive to the robots.txt ?

Reply With Quote
  #2  
Old January 17th, 2005, 04:11 PM
jharnois's Avatar
jharnois jharnois is offline
mod_dev_shed
Dev Shed God 19th Plane (14000 - 14499 posts)
 
Join Date: Sep 2002
Location: Atlanta, GA
Posts: 14,295 jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level) 
Time spent in forums: 1 Month 1 Week 2 Days 1 h 36 m 24 sec
Reputation Power: 760
The advantage to using robots.txt is that the information is all in one file. This is similar to using external CSS; all the style is in one file. You can quickly change the spider policies by changing only one file. If you don't have one at all you'll get errors in your log files (if you care). However, in your case, I might be inclined to use only the meta (provided I didn't care about the error logs) since the policy is only in one page anyway. I usually use both, and either will work.

Reply With Quote
  #3  
Old January 18th, 2005, 11:26 AM
frankieluccini's Avatar
frankieluccini frankieluccini is offline
Contributing User
Dev Shed Newbie (0 - 499 posts)
 
Join Date: Jan 2004
Posts: 374 frankieluccini User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: 2 Days 9 h 4 m 53 sec
Reputation Power: 5
thanks for the reply.
with that said.........
is there any disadvantage in including both the meta and
the entire robots.txt ?
do some spiders use one and others use the other ?
have you ever seen any conflicts when using both ?

thanks again.

Reply With Quote
  #4  
Old January 18th, 2005, 02:30 PM
jharnois's Avatar
jharnois jharnois is offline
mod_dev_shed
Dev Shed God 19th Plane (14000 - 14499 posts)
 
Join Date: Sep 2002
Location: Atlanta, GA
Posts: 14,295 jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level)jharnois User rank is Major General (70000 - 90000 Reputation Level) 
Time spent in forums: 1 Month 1 Week 2 Days 1 h 36 m 24 sec
Reputation Power: 760
I've never had any issues using both. Things I want indexed seem to be on my sites, and those I don't don't seem to be. However, each spider will handle the conflict differently. Some may place more weight on robots.txt while others on meta. I'm sure Google explains their practice somewhere.
__________________
# Jeremy

Explain your problem instead of asking how to do what you decided was the solution.

Reply With Quote
Reply

Viewing: Dev Shed ForumsWeb DesignSearch Engine Optimization > robots_txt vs robots meta


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 5 hosted by Hostway