#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    3
    Rep Power
    0

    NOAA urllib and csv


    Hello,
    So according to the title, I am trying to grab stuff from a URL, and save the data as a csv file.
    I have it working nicely when I do print, but not the actuall writing to the csv file.
    This is my code now:

    Code:
    import urllib2
    from bs4 import BeautifulSoup
    from time import localtime, strftime
    import csv
    
    #This script is intended to retrive NOAA data and apend it to a csv file.
    # for time inormation http://docs.python.org/library/time.html
    
    # Wait 45 min
    #Need to work on this part...
    
    
    # Go into URL
    page = urllib2.urlopen("http://w1.weather.gov/obhistory/PAFA.html")
    soup = BeautifulSoup(page)
    datemonth=strftime("%m", localtime())
    dateday=strftime("%d", localtime())
    
    
    
    with open("/home/eyalak/Documents/weather/weather.csv", "wb") as f:
          writer = csv.writer(f)
          table = soup.findAll("table")[3]
          #print table
          for tr in table.findAll("tr", valign="top"):
             a={x.string for x in tr.findAll('td')}
             print str(a)
             writer.writerows([a])
    The output is:
    Code:
    >>> 
    set([u'29.84', u'09', u'16', u'OVC080', u'NA', None, u'Overcast', u'1011.8', u'71%', u'Calm', u'8', u'18:53', u'10.00'])
    Traceback (most recent call last):
      File "/home/eyalak/Documents/weather/weather_v0.py", line 28, in <module>
        writer.writerows([a])
    Error: sequence expected
    It is important to notice that the "print str(a) " prints only the last line, or it is overwriten and ofcourse the error message that I have not figured out yet.
    Thanks for any help
    Eyal
  2. #2
  3. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481
    Originally Posted by http://docs.python.org/3/library/csv.html#writer-objects
    A row must be a sequence of strings or numbers for Writer objects
    Originally Posted by http://docs.python.org/3/library/index.html
    4.6. Sequence Types list, tuple, range
    You passed a set
    {1,2,3}
    where python requires a sequence.

    Where you have
    a={x.string for x in tr.findAll('td')}

    use
    a=list({x.string for x in tr.findAll('td')})

    Sets are unordered. You might want to sort or organize a.


    Your delay:

    import time
    time.sleep(45*60)
    [code]Code tags[/code] are essential for python code and Makefiles!
  4. #3
  5. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    3
    Rep Power
    0

    I do not have a way to sort it out. I would like the output to come out in the order.


    Thnanks for you help. But the output are numbers that I do not want to altered by their value, but by the order of the column in the HTML table the data comes from...

    list() just makes a mess..

    Originally Posted by b49P23TIvg
    You passed a set
    {1,2,3}
    where python requires a sequence.

    Where you have
    a={x.string for x in tr.findAll('td')}

    use
    a=list({x.string for x in tr.findAll('td')})

    Sets are unordered. You might want to sort or organize a.


    Your delay:

    import time
    time.sleep(45*60)
  6. #4
  7. Contributing User
    Devshed Demi-God (4500 - 4999 posts)

    Join Date
    Aug 2011
    Posts
    4,995
    Rep Power
    481
    Originally Posted by whitepython
    the error message that I have not figured out yet.
    I haven't yet had to learn about beautiful soup. I merely tried to explain that your "row" is invalid because you have a set type where python requires a sequence. Furthermore, I cited reading references for the issues.
    [code]Code tags[/code] are essential for python code and Makefiles!
  8. #5
  9. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Nov 2012
    Posts
    3
    Rep Power
    0

    csv sequence


    I do not understand why data input to CSV must be as a list().

    Csv files can have various numbers in each coumn. If in each row the columns will be organized by the order priority of a number (1....10) thos will cause a mess...

    I can not organize the list based on the values in it, since they are constantly changing...
    Thanks
    E

IMN logo majestic logo threadwatch logo seochat tools logo