#1
  1. Web Developer
    Devshed Novice (500 - 999 posts)

    Join Date
    Oct 2001
    Location
    Finland
    Posts
    719
    Rep Power
    16

    How to make robot.txt


    Can anyone tell me how robot.txt works and how to make one?
    -- Tomi Kaistila
    -- Developer's Journal

    The more you learn, the more you know.
    The more you know, the more you forget.
    The more you forget, the less you know.
  2. #2
  3. No Profile Picture
    Python Prophet
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2001
    Location
    Amersfoort, The Netherlands
    Posts
    45
    Rep Power
    14
    Take a look at the official website.
    Some people, when confronted with a problem, think I know, I'll use regular expressions. Now they have two problems. - Jamie Zawinski, in comp.lang.emacs

IMN logo majestic logo threadwatch logo seochat tools logo