April 27th, 2004, 05:32 PM
Big file unpickling question
Here's my problem:
I've got a big (350MB) file, composed of a lot of variable-sized pickled objects. Leaving aside the question of how this situation presented itself, is there a quick, easy and elegant way of getting the last record?
I don't want to read the whole lot from the start, and suffer the overhead, or seek to the end, rewind a fixed distance, throw away the partial record, and read to the end, because that's not portable for other object types (which I suspect I will need) without tweaking the rewind distance.
April 29th, 2004, 08:32 PM
I would split the pickles into separate files, and create a directory file, telling the date, name, etc of the data in each pickle file.