#1
  1. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2000
    Posts
    12
    Rep Power
    0

    Question Internet Spiders


    Does anyone know how an Internet Spider works? I understand the whole concept that the spider goes out and "crawls" the internet sites and looks for meta tag, html code, and things of that nature. I know that it follows links from one page to another, depending on how deep the "spider" software is told to search. My question is how does it know where to search? How does it acutally find the pages?
  2. #2
  3. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2000
    Posts
    23
    Rep Power
    0

    seed it


    you seed the spider with a list of url's to attack. then it adds to the list itself as it traverses pages (usually only hitting content-types of html or text).

    the tricky parts come in revisiting pages and updating the indexes that result from collating keywords. to do it right, spiders are written in C/C++ for highest performance. lots of open-source and commercial products out there to do spidering.


    ===========================================
    http://badblue.com/helpphp.htm
    Free small footprint web server for Windows
    PHP, P2P file-sharing, transcoding and more
    ===========================================
  4. #3
  5. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2000
    Posts
    12
    Rep Power
    0
    Do you seed it with a range of IP addresses? I don't actually have to code this. I'm just trying to understand the whole process better.
  6. #4
  7. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Dec 2000
    Posts
    23
    Rep Power
    0

    found one


    Found a php based spider... no idea how good it is...

    http://phpdig.toiletoine.net/


    ===========================================
    http://badblue.com/helpphp.htm
    Free small footprint web server for Windows
    PHP, P2P file-sharing, transcoding and more
    ===========================================

IMN logo majestic logo threadwatch logo seochat tools logo