April 27th, 2013, 11:07 AM
Generate Unique Links
Is there away to generate a list of unique links within a website?
I have a script that processes webpages but I want to feed it the links.
April 27th, 2013, 04:04 PM
Not without making a spider.
I think the best way with what you have would be to modify the processor so that it also captures links on a page, then record those links somewhere, pick one to visit, flag it as "visited" so you don't hit it again, and repeat.
April 27th, 2013, 06:27 PM
I was hoping to find something already written to do this.
I will work on a script that will do this. The user will enter a domain name and the script will crawl the site and create an array of unique URLs.
I will report here when is is done for others to comment and perhaps copy.