#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2016
    Location
    London
    Posts
    3
    Rep Power
    0

    Which is better Robots.txt or Meta Robots Tag?


    Hello friends,

    Which is better Robots.txt or Meta Robots Tag?
  2. #2
  3. Lazy Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    16,333
    Rep Power
    9645
    + robots.txt can stop a robot from attempting to crawl certain URLs
    - robots.txt has to have each URL to block listed inside, with some support for wildcards

    + <meta> tag can be customized for each page
    + <meta> tag offers more detailed instructions about whether to allow indexing or to allow following links
    - <meta> tag has to be managed on each page
    - <meta> tag requires the bot to crawl the page so it cannot prevent that from happening

IMN logo majestic logo threadwatch logo seochat tools logo