1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Sep 2013
    Rep Power

    Extracting data from password protected page

    Hi Guys, hope your all well

    Im looking to try and make a script that will go to a password protected website (login) if possible, then extract data from that page and store it in MySQL database.

    Two things I am wondering if you don't mind giving the advice is

    1. How would I make the script automatically login?
    2. How can I make the script go to the next page and keep going through every page within that section.

    the page navigation is displayed like

    1 | 2| 3 | 4 | 5 | and so on... (different number or pages in each section)
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Loyal (3000 - 3499 posts)

    Join Date
    Dec 2004
    Rep Power
    there is a sticky post at the top
  4. #3
  5. Sarcky
    Devshed Supreme Being (6500+ posts)

    Join Date
    Oct 2006
    Pennsylvania, USA
    Rep Power
    The sticky is for making your own password-protected page, not stealing content off someone else's.

    Since this is usually illegal, I'm going to give a couple of hints and then close the thread.

    Snoopy is a PHP class which masquerades as a browser and can do form posts and cookie control.

    The DOMDocument functions allow you to crawl an HTML document as an object tree

    Regular expressions can be used to operate on the whole document.

    Store local copies of every page you fetch so you don't have to re-fetch and set off security alarms.

    Thread closed, contact a lawyer before you do this.

    Comments on this post

    • paulh1983 agrees : sorry, i didnt read his post properly:(
    HEY! YOU! Read the New User Guide and Forum Rules

    "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety." -Benjamin Franklin

    "The greatest tragedy of this changing society is that people who never knew what it was like before will simply assume that this is the way things are supposed to be." -2600 Magazine, Fall 2002

    Think we're being rude? Maybe you asked a bad question or you're a Help Vampire. Trying to argue intelligently? Please read this.

IMN logo majestic logo threadwatch logo seochat tools logo