#1
  1. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2005
    Posts
    37
    Rep Power
    10

    Getting data from web sites requiring cookies


    I am very new to web programming, and semi-new to python. Here's what I'm trying to do, though...
    I would like to access data from yahoo finance, namely real time stock market data. I will then take this data and feed it to a neural net I've developed, looking for patterns. My problem is that I have not been able to use either urllib or httplib to open the appropriate web page such that I can get at the data. I have searched within this forum and have read many helpful tips, but so far nothing has worked for me.

    From my browser, I can simply open the correct page, but when using python to access the same url, I get the login screen since I have not figured out how to send the appropriate cookie along to yahoo.

    I have not been able to post the login information to the login page either, since it seems yahoo hashes the password before posting, and cracking this nut is beyond my web programming abilities so far.

    Any help would be greatly appreciated, (and if my longshot get-rich-quick scheme works, rewarded )
  2. #2
  3. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jan 2005
    Posts
    37
    Rep Power
    10

    Smile Follow up - problem solved


    In case anyone finds this thread in a search...
    I used Scorpions4ever's wonderful IEC library. This method circumvented the problem I was having by using the browser directly, and hence the proper cookies. When I get smarter, I'll figure out a cleaner way to deal with cookies, but until then Scorpions saved the day for me. Thanks!

    PS, I am still looking for that cleaner way, so tips would be welcome!
  4. #3
  5. No Profile Picture
    Contributing User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Jun 2004
    Posts
    461
    Rep Power
    25
    i have played with urllib2 module for python and it seemed to handle recieving cookies. I never got to indepth with it, so i am not sure if it will work for you, maybe you should give it a try

IMN logo majestic logo threadwatch logo seochat tools logo