January 3rd, 2005, 06:17 PM
Getting data from web sites requiring cookies
I am very new to web programming, and semi-new to python. Here's what I'm trying to do, though...
I would like to access data from yahoo finance, namely real time stock market data. I will then take this data and feed it to a neural net I've developed, looking for patterns. My problem is that I have not been able to use either urllib or httplib to open the appropriate web page such that I can get at the data. I have searched within this forum and have read many helpful tips, but so far nothing has worked for me.
From my browser, I can simply open the correct page, but when using python to access the same url, I get the login screen since I have not figured out how to send the appropriate cookie along to yahoo.
I have not been able to post the login information to the login page either, since it seems yahoo hashes the password before posting, and cracking this nut is beyond my web programming abilities so far.
Any help would be greatly appreciated, (and if my longshot get-rich-quick scheme works, rewarded )
January 3rd, 2005, 08:10 PM
Follow up - problem solved
In case anyone finds this thread in a search...
I used Scorpions4ever's wonderful IEC library. This method circumvented the problem I was having by using the browser directly, and hence the proper cookies. When I get smarter, I'll figure out a cleaner way to deal with cookies, but until then Scorpions saved the day for me. Thanks!
PS, I am still looking for that cleaner way, so tips would be welcome!
January 10th, 2005, 11:33 AM
i have played with urllib2 module for python and it seemed to handle recieving cookies. I never got to indepth with it, so i am not sure if it will work for you, maybe you should give it a try