#1
  1. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Nov 2003
    Posts
    701
    Rep Power
    96

    Generate Unique Links


    Is there away to generate a list of unique links within a website?

    I have a script that processes webpages but I want to feed it the links.

    Thanks,
    Evan
  2. #2
  3. Transforming Moderator
    Devshed Supreme Being (6500+ posts)

    Join Date
    Mar 2007
    Location
    Washington, USA
    Posts
    14,294
    Rep Power
    9400
    Not without making a spider.

    I think the best way with what you have would be to modify the processor so that it also captures links on a page, then record those links somewhere, pick one to visit, flag it as "visited" so you don't hit it again, and repeat.
  4. #3
  5. Contributing User
    Devshed Novice (500 - 999 posts)

    Join Date
    Nov 2003
    Posts
    701
    Rep Power
    96
    I was hoping to find something already written to do this.

    I will work on a script that will do this. The user will enter a domain name and the script will crawl the site and create an array of unique URLs.

    I will report here when is is done for others to comment and perhaps copy.
    Evan

IMN logo majestic logo threadwatch logo seochat tools logo