#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    13
    Rep Power
    0

    Question Prevent crawlers from submitting forms using htaccess


    Crawlers like Google submit forms to try to get more pages to index. I heard there is way to prevent crawlers from submitting form using htaccess and robots.txt.

    Do you know how to do it using htaccess?
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Location
    Saint-Petersburg, Russia
    Posts
    236
    Rep Power
    28
    That means that if a search form is forbidden in robots.txt, we won't crawl any of the URLs that a form would generate. Similarly, we only retrieve GET forms and avoid forms that require any kind of user information.
    So obviously you can either put url of form's action to robots.txt or change your form to POST method.

    http://googlewebmastercentral.blogsp...tml-forms.html

    also I think you can use googleoff comments.
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    13
    Rep Power
    0
    but there is an actual rule to achieve this but can't remember it at all
  6. #4
  7. Transforming Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,111
    Rep Power
    9398
    Did you read that link? There are a couple things they specifically mention.
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2013
    Posts
    13
    Rep Power
    0
    They mention that it's an experiment but the article if from 2008 so i think it's already active now from long time. If it's not active, how some forums have search queries indexed?

IMN logo majestic logo threadwatch logo seochat tools logo