#1
  1. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2004
    Posts
    1
    Rep Power
    0

    Big file unpickling question


    Here's my problem:

    I've got a big (350MB) file, composed of a lot of variable-sized pickled objects. Leaving aside the question of how this situation presented itself, is there a quick, easy and elegant way of getting the last record?

    I don't want to read the whole lot from the start, and suffer the overhead, or seek to the end, rewind a fixed distance, throw away the partial record, and read to the end, because that's not portable for other object types (which I suspect I will need) without tweaking the rewind distance.

    Any ideas?
  2. #2
  3. No Profile Picture
    Registered User
    Devshed Newbie (0 - 499 posts)

    Join Date
    Apr 2003
    Posts
    25
    Rep Power
    0
    I would split the pickles into separate files, and create a directory file, telling the date, name, etc of the data in each pickle file.

IMN logo majestic logo threadwatch logo seochat tools logo