December 21st, 2012, 09:26 PM
Scraping by date
say I have a website "website.com" and the date is at the end of the url in the form 20121221(iow YearMonthDay). So Dec 21, 2012 would be "website.com/20121221"
I want to perform a few scraping functions for each day, let's call these functions "a" and "b." I have no problem doing that for any 1 day in particular. But I want my program to do it for today, then yesterday, then the day before, etc. All the way back to 3 years ago. How do I do this?
When I get all of the info I want, how do I import it directly from command line to my database in sql?
Thank you for any help.
December 22nd, 2012, 12:16 AM
Python has a datetime module specifically to do date and time calculations:
# Change start_date and end_date as needed
start_date = datetime.date(2012, 12, 21)
end_date = datetime.date(2012, 12, 1)
while start_date >= end_date:
date_str = start_date.strftime("%Y%m%d")
# Do something here with date_str like concat to your URL
start_date = start_date - datetime.timedelta(days=1)
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne
"I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo