Thread: Robots.txt

    #1
  1. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Posts
    161
    Rep Power
    32

    Exclamation Robots.txt


    User-agent: *
    Allow: /
    Allow: /my/announcements*
    Allow: /my/knowledgebase*
    Allow: /my/downloads*
    Disallow: /my/
    Disallow: /images/
    Disallow: /includes/
    SITEMAP: http://www.option9.com/sitemap.xml

    I would like to not permit crawlers access to subdomain my with the exception of announcements, knowledgebase and downloads.

    I need the crawlers to have access to the above three if they act as directory or .php. For instance the url could be my.option9.com/announcements.php or it could be my.option9.com/announcements/example.php

    Did I place the asteric correctly for wildcard purposes? As well is the disallow statement after my allows going to interfere?
    Free Domain Names || Free Web Hosting
  2. #2
  3. Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2003
    Posts
    161
    Rep Power
    32
    Does anyone know the answer to the above asked questions which have been posted by Nullified on above post's stated date?
    Free Domain Names || Free Web Hosting

IMN logo majestic logo threadwatch logo seochat tools logo