Thread: Robots.txt

    #1
  1. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Posts
    159
    Rep Power
    31

    Exclamation Robots.txt


    User-agent: *
    Allow: /
    Allow: /my/announcements*
    Allow: /my/knowledgebase*
    Allow: /my/downloads*
    Disallow: /my/
    Disallow: /images/
    Disallow: /includes/
    SITEMAP: http://www.option9.com/sitemap.xml

    I would like to not permit crawlers access to subdomain my with the exception of announcements, knowledgebase and downloads.

    I need the crawlers to have access to the above three if they act as directory or .php. For instance the url could be my.option9.com/announcements.php or it could be my.option9.com/announcements/example.php

    Did I place the asteric correctly for wildcard purposes? As well is the disallow statement after my allows going to interfere?
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Posts
    159
    Rep Power
    31
    Does anyone know the answer to the above asked questions which have been posted by Nullified on above post's stated date?

IMN logo majestic logo threadwatch logo seochat tools logo