|
|
|||||||||
|
|||||||||
| |||||||||
|
|
|
| |||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
#1
|
||||
|
||||
|
robots_txt vs robots meta
is there any advantage in using a robots.txt file opposed
to just including a robots meta tag ? it seems that there may be, if you want the spider to view specific sub-directories and avoid others, but im only trying to focus on the index page of my site. ...... index, nofollow ....... can anyone provide any input for me? should i be ok with only including the meta, or are the spiders more perceptive to the robots.txt ? |
|
#2
|
||||
|
||||
|
The advantage to using robots.txt is that the information is all in one file. This is similar to using external CSS; all the style is in one file. You can quickly change the spider policies by changing only one file. If you don't have one at all you'll get errors in your log files (if you care). However, in your case, I might be inclined to use only the meta (provided I didn't care about the error logs) since the policy is only in one page anyway. I usually use both, and either will work.
|
|
#3
|
||||
|
||||
|
thanks for the reply.
with that said......... is there any disadvantage in including both the meta and the entire robots.txt ? do some spiders use one and others use the other ? have you ever seen any conflicts when using both ? thanks again. |
|
#4
|
||||
|
||||
|
I've never had any issues using both. Things I want indexed seem to be on my sites, and those I don't don't seem to be. However, each spider will handle the conflict differently. Some may place more weight on robots.txt while others on meta. I'm sure Google explains their practice somewhere.
__________________
# Jeremy Explain your problem instead of asking how to do what you decided was the solution. |
![]() |
| Viewing: Dev Shed Forums > Web Design > Search Engine Optimization > robots_txt vs robots meta |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|