#1
  1. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2003
    Posts
    3
    Rep Power
    0

    Web crawler in javascript


    I'm trying to write a simple web crawler in Javascript. The basic idea is very simple:

    - load a page into a separate window
    - append the links to a list of pending links
    - load every page that is linked to in turn adding the links to the pending list if they haven't been visited.
    - keep going until you run out of links.

    My problem is that I can't figure out how to tell when the page has finished loading. I tried adding an onload handler to the window in which I load the page but it didn't run.

    Any ideas?

    Kevin Whitefoot
  2. #2
  3. Second highest poster :p
    Devshed Supreme Being (6500+ posts)

    Join Date
    Jul 2001
    Posts
    7,322
    Rep Power
    33
    have you tried placing the section that calls the script at the very end of the page?
    - Andreas Koepke

    Koepke Photography

  4. #3
  5. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2003
    Posts
    3
    Rep Power
    0
    Just in case anyone is interested the answer is to use document.readystate.

    You can see the result on my web site at:

    http://home.c2i.net/kwhitefoot/JsCrawler.html

IMN logo majestic logo threadwatch logo seochat tools logo