Scraping by date - Dev Shed

Dev Shed Forums Sponsor:

December 21st, 2012, 09:26 PM

ThanksABunch

Registered User

Join Date: Mar 2012

Posts: 2

Time spent in forums: 36 m 46 sec
Reputation Power: 0

Scraping by date

say I have a website "website.com" and the date is at the end of the url in the form 20121221(iow YearMonthDay). So Dec 21, 2012 would be "website.com/20121221"

I want to perform a few scraping functions for each day, let's call these functions "a" and "b." I have no problem doing that for any 1 day in particular. But I want my program to do it for today, then yesterday, then the day before, etc. All the way back to 3 years ago. How do I do this?

*BONUS*

When I get all of the info I want, how do I import it directly from command line to my database in sql?

Thank you for any help.

December 22nd, 2012, 12:16 AM

Scorpions4ever

Banned ;)

Dev Shed God 9th Plane (9000 - 9499 posts)

Join Date: Nov 2001

Location: Woodland Hills, Los Angeles County, California, USA

Posts: 9,383

Time spent in forums: 1 Month 4 Weeks 1 Day 20 h 49 m 30 sec
Reputation Power: 4080

Python has a datetime module specifically to do date and time calculations:

python Code:

Original - python Code
import datetime # Change start_date and end_date as needed start_date = datetime.date(2012, 12, 21) end_date = datetime.date(2012, 12, 1) while start_date >= end_date: date_str = start_date.strftime("%Y%m%d") # Do something here with date_str like concat to your URL print(date_str) start_date = start_date - datetime.timedelta(days=1)

 
import datetime
 
# Change start_date and end_date as needed
start_date = datetime.date(2012, 12, 21)
end_date = datetime.date(2012, 12, 1)  
 
while start_date >= end_date:
    date_str = start_date.strftime("%Y%m%d")
    # Do something here with date_str like concat to your URL
    print(date_str)
    start_date = start_date - datetime.timedelta(days=1)

__________________
Up the Irons
What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
"Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
Down with Sharon Osbourne

Viewing: Dev Shed Forums > Programming Languages > Python Programming > Scraping by date

« Previous Thread | Next Thread »

Developer Shed Advertisers and Affiliates

Thread Tools	Search this Thread
Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread:

Posting Rules
You may not post new threads You may not post replies You may not post attachments You may not edit your posts vB code is On Smilies are On [IMG] code is On HTML code is Off

View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox

Forum Jump