Problem with cPickle Files

Okay I have a web site that is run entirely by Python. All data is stored as Pickled python classes. The problem is from time to time the files will get truncated. Sometimes they go to zero bytes and others they will be cut in size as though the write process was interrupted. No error is raised until the next time I try to load the pickle data and I get an EOFError. What could be causing this? I was thinking file locking might be a good idea but the documnenation of file locking is poor. I was able to lock a file on NT but I don't know how to lock a file on Linux via Python code. I am not entirly certain file locking is the problem. Has anyone else had a simular problem? Here is an example of one of the save data functions.

import cPickle
characterDir = '../db/chars'

def saveCharacter(charObj):
if not isinstance(charObj,Character):
raise TypeError("Expected type Character")
fo = open(characterDir + '/' + charObj.name.lower(),'wb')
cPickle.dump(charObj,fo,1)

I surpose its fesable that locking is the problem, if two users tried to write to the Pickled file at the same time the data could be corrupted - whenever i've worked with cPickle i i've tried to store the data in the form of a dictionary for easy access - I cant say say that i've ever had your problem before.. can't see anything wrong with your function either.. is this the only part of your script which adds to the Pickle?

Mark.

Well the thing is their are multiple scripts that saved different kinds of files. Each of them have problems randomly from time to time. It seems to be pretty rare when it happens. I am going to be moving to another server soon so maybe that will help. Does anyone have an example for locking a file under a Linux system? Can you give me more information on storing your data in the form of a dictionary?

Here it is - a very basic example of how to store a dictionary inside a pickled file and read it back into the program - wheather or not this will fix the problem with the picked files getting cut up i don't know!



I will look into file locking for you..

Mark.

Yeah thats basically what I have for my code. So it must either be file locking or something wrong with the server. If anyone has file locking code for Linux, I would appreciate it.

Ok a few links for you, the first is a module called 'fcntl' which seems like the right way to go about file locking in *nix.. the second is a script i found in ASPNs Python Cookbook which can lock files on Win32 and *nix:

http://www.python.org/doc/current/lib/module-fcntl.html
http://aspn.activestate.com/ASPN/Co...on/Recipe/65203

Mark.

Thank you very much! This really helps.