October 25th, 2013, 04:10 PM
Extracting data from password protected page
Hi Guys, hope your all well
Im looking to try and make a script that will go to a password protected website (login) if possible, then extract data from that page and store it in MySQL database.
Two things I am wondering if you don't mind giving the advice is
1. How would I make the script automatically login?
2. How can I make the script go to the next page and keep going through every page within that section.
the page navigation is displayed like
1 | 2| 3 | 4 | 5 | and so on... (different number or pages in each section)
October 25th, 2013, 06:29 PM
there is a sticky post at the top
October 25th, 2013, 07:03 PM
The sticky is for making your own password-protected page, not stealing content off someone else's.
Since this is usually illegal, I'm going to give a couple of hints and then close the thread.
Snoopy is a PHP class which masquerades as a browser and can do form posts and cookie control.
The DOMDocument functions allow you to crawl an HTML document as an object tree
Regular expressions can be used to operate on the whole document.
Store local copies of every page you fetch so you don't have to re-fetch and set off security alarms.
Thread closed, contact a lawyer before you do this.
Comments on this post
HEY! YOU! Read the New User Guide and Forum Rules
"They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin
"The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002
Think we're being rude? Maybe you asked a bad question
or you're a Help Vampire.
Trying to argue intelligently? Please read this.