1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Mar 2012
    Rep Power

    Scraping by date

    say I have a website "website.com" and the date is at the end of the url in the form 20121221(iow YearMonthDay). So Dec 21, 2012 would be "website.com/20121221"

    I want to perform a few scraping functions for each day, let's call these functions "a" and "b." I have no problem doing that for any 1 day in particular. But I want my program to do it for today, then yesterday, then the day before, etc. All the way back to 3 years ago. How do I do this?


    When I get all of the info I want, how do I import it directly from command line to my database in sql?

    Thank you for any help.
  2. #2
  3. Banned ;)
    Devshed Supreme Being (6500+ posts)

    Join Date
    Nov 2001
    Woodland Hills, Los Angeles County, California, USA
    Rep Power
    Python has a datetime module specifically to do date and time calculations:
    python Code:
    import datetime
    # Change start_date and end_date as needed
    start_date = datetime.date(2012, 12, 21)
    end_date = datetime.date(2012, 12, 1)  
    while start_date >= end_date:
        date_str = start_date.strftime("%Y%m%d")
        # Do something here with date_str like concat to your URL
        start_date = start_date - datetime.timedelta(days=1)
    Up the Irons
    What Would Jimi Do? Smash amps. Burn guitar. Take the groupies home.
    "Death Before Dishonour, my Friends!!" - Bruce D ickinson, Iron Maiden Aug 20, 2005 @ OzzFest
    Down with Sharon Osbourne

    "I wouldn't hire a butcher to fix my car. I also wouldn't hire a marketing firm to build my website." - Nilpo

IMN logo majestic logo threadwatch logo seochat tools logo