Problem with cPickle Files

Dev Shed Forums Sponsor:

October 28th, 2003, 08:25 PM

AzulSorcerer

Junior Member

Join Date: Oct 2003

Location: CA

Posts: 5

Time spent in forums: < 1 sec
Reputation Power: 0

Problem with cPickle Files

Okay I have a web site that is run entirely by Python. All data is stored as Pickled python classes. The problem is from time to time the files will get truncated. Sometimes they go to zero bytes and others they will be cut in size as though the write process was interrupted. No error is raised until the next time I try to load the pickle data and I get an EOFError. What could be causing this? I was thinking file locking might be a good idea but the documnenation of file locking is poor. I was able to lock a file on NT but I don't know how to lock a file on Linux via Python code. I am not entirly certain file locking is the problem. Has anyone else had a simular problem? Here is an example of one of the save data functions.

import cPickle
characterDir = '../db/chars'

def saveCharacter(charObj):
if not isinstance(charObj,Character):
raise TypeError("Expected type Character")
fo = open(characterDir + '/' + charObj.name.lower(),'wb')
cPickle.dump(charObj,fo,1)

October 29th, 2003, 02:49 AM

netytan

Hello World :)

Join Date: Mar 2003

Location: Hull, UK

Posts: 2,537

Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68

I surpose its fesable that locking is the problem, if two users tried to write to the Pickled file at the same time the data could be corrupted - whenever i've worked with cPickle i i've tried to store the data in the form of a dictionary for easy access - I cant say say that i've ever had your problem before.. can't see anything wrong with your function either.. is this the only part of your script which adds to the Pickle?

Mark.

__________________
programming language development: www.netytan.com – Hula

October 29th, 2003, 03:14 AM

AzulSorcerer

Junior Member

Join Date: Oct 2003

Location: CA

Posts: 5

Time spent in forums: < 1 sec
Reputation Power: 0

Well the thing is their are multiple scripts that saved different kinds of files. Each of them have problems randomly from time to time. It seems to be pretty rare when it happens. I am going to be moving to another server soon so maybe that will help. Does anyone have an example for locking a file under a Linux system? Can you give me more information on storing your data in the form of a dictionary?

October 29th, 2003, 11:57 AM

netytan

Hello World :)

Join Date: Mar 2003

Location: Hull, UK

Posts: 2,537

Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68

Here it is - a very basic example of how to store a dictionary inside a pickled file and read it back into the program - wheather or not this will fix the problem with the picked files getting cut up i don't know!

Code:

#!/usr/bin/env python

import cPickle

#some dictionary that you wish to store for later, this could contain
#other dicrionaries, lists, tuples, strings etc.

dictionary1 = {'one': 1, 'two': 2, 'three': 3}

#save the dictionary object in the pickle file..
cPickle.dump(dictionary1, file('pickle.db', 'w'))

#read in the pickled dictionary and assign it to a variable! you can now use
#it like any other dictionary.

dictionary2 = cPickle.load(file('pickle.db', 'r'))

#do something with the dictionary.
print dictionary2

I will look into file locking for you..

Mark.

October 29th, 2003, 12:36 PM

AzulSorcerer

Junior Member

Join Date: Oct 2003

Location: CA

Posts: 5

Time spent in forums: < 1 sec
Reputation Power: 0

Yeah thats basically what I have for my code. So it must either be file locking or something wrong with the server. If anyone has file locking code for Linux, I would appreciate it.

October 29th, 2003, 02:21 PM

netytan

Hello World :)

Join Date: Mar 2003

Location: Hull, UK

Posts: 2,537

Time spent in forums: 1 Week 2 Days 18 h 17 m 47 sec
Reputation Power: 68

Ok a few links for you, the first is a module called 'fcntl' which seems like the right way to go about file locking in *nix.. the second is a script i found in ASPNs Python Cookbook which can lock files on Win32 and *nix:

http://www.python.org/doc/current/lib/module-fcntl.html
http://aspn.activestate.com/ASPN/Co...on/Recipe/65203

Mark.

October 29th, 2003, 04:41 PM

AzulSorcerer

Junior Member

Join Date: Oct 2003

Location: CA

Posts: 5

Time spent in forums: < 1 sec
Reputation Power: 0

Thank you very much! This really helps.

Problem with cPickle Files

Developer Shed Advertisers and Affiliates