#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2001
    Location
    entlegen
    Posts
    442
    Rep Power
    13

    can u show me how to stop him in "iptables" ??????????


    4-0 5254 0/91/91 _ 0.06 11036 0 0.0 0.48 0.48 66.28.98.5 www.myserver.com GET /robots.txt HTTP/1.0
    5-0 5255 0/105/105 _ 0.06 9369 0 0.0 0.28 0.28 66.28.98.4 www.myserver.com GET /robots.txt HTTP/1.0

    this is taken from the same status report.

    can u show me how to stop him in "iptables" ??????????
    Last edited by pentium5; March 12th, 2002 at 07:04 AM.
    regards,

    wish to use AT 89c51 single chip computer to do remote sensing and send the data back from woods to office via nokie 3330 and internet. HOW?
    first I must revise C program thro' example, what's next?
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed God 1st Plane (5500 - 5999 posts)

    Join Date
    Oct 2000
    Location
    Back in the real world.
    Posts
    5,966
    Rep Power
    190
    why do you want to stop him? seems like a crawler from a search engine and if it only requests robots.txt, this is fine, it complies to the standards.

    give it a "robots.txt" with:

    User-agent: *
    Disallow: /

    and it won´t bother you...

    there is 10000s like this on the web, so you need to block quite many IPs. (i know of altavista having at least 100 different ones that showed up in my server logs already)
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2001
    Posts
    4
    Rep Power
    0
    >> why do you want to stop him?

    Good question. Same question that we have asked pentium5 dozen (100+) times without a valid reason in doing so.

    Maybe he should setup some kind of access restriction in his docroot and have a default-to-deny policy.

IMN logo majestic logo threadwatch logo seochat tools logo