#1
  1. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Location
    CA
    Posts
    5
    Rep Power
    0

    Question Problem with cPickle Files


    Okay I have a web site that is run entirely by Python. All data is stored as Pickled python classes. The problem is from time to time the files will get truncated. Sometimes they go to zero bytes and others they will be cut in size as though the write process was interrupted. No error is raised until the next time I try to load the pickle data and I get an EOFError. What could be causing this? I was thinking file locking might be a good idea but the documnenation of file locking is poor. I was able to lock a file on NT but I don't know how to lock a file on Linux via Python code. I am not entirly certain file locking is the problem. Has anyone else had a simular problem? Here is an example of one of the save data functions.

    import cPickle
    characterDir = '../db/chars'

    def saveCharacter(charObj):
    if not isinstance(charObj,Character):
    raise TypeError("Expected type Character")
    fo = open(characterDir + '/' + charObj.name.lower(),'wb')
    cPickle.dump(charObj,fo,1)
  2. #2
  3. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    I surpose its fesable that locking is the problem, if two users tried to write to the Pickled file at the same time the data could be corrupted - whenever i've worked with cPickle i i've tried to store the data in the form of a dictionary for easy access - I cant say say that i've ever had your problem before.. can't see anything wrong with your function either.. is this the only part of your script which adds to the Pickle?

    Mark.
    programming language development: www.netytan.com Hula

  4. #3
  5. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Location
    CA
    Posts
    5
    Rep Power
    0
    Well the thing is their are multiple scripts that saved different kinds of files. Each of them have problems randomly from time to time. It seems to be pretty rare when it happens. I am going to be moving to another server soon so maybe that will help. Does anyone have an example for locking a file under a Linux system? Can you give me more information on storing your data in the form of a dictionary?
  6. #4
  7. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Here it is - a very basic example of how to store a dictionary inside a pickled file and read it back into the program - wheather or not this will fix the problem with the picked files getting cut up i don't know!

    Code:
    #!/usr/bin/env python
    
    import cPickle
    
    #some dictionary that you wish to store for later, this could contain
    #other dicrionaries, lists, tuples, strings etc.
    
    dictionary1 = {'one': 1, 'two': 2, 'three': 3}
    
    #save the dictionary object in the pickle file..
    cPickle.dump(dictionary1, file('pickle.db', 'w'))
    
    #read in the pickled dictionary and assign it to a variable! you can now use
    #it like any other dictionary.
    
    dictionary2 = cPickle.load(file('pickle.db', 'r'))
    
    #do something with the dictionary.
    print dictionary2
    I will look into file locking for you..

    Mark.
    programming language development: www.netytan.com Hula

  8. #5
  9. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Location
    CA
    Posts
    5
    Rep Power
    0
    Yeah thats basically what I have for my code. So it must either be file locking or something wrong with the server. If anyone has file locking code for Linux, I would appreciate it.
  10. #6
  11. Hello World :)
    Devshed Frequenter (2500 - 2999 posts)

    Join Date
    Mar 2003
    Location
    Hull, UK
    Posts
    2,537
    Rep Power
    69
    Ok a few links for you, the first is a module called 'fcntl' which seems like the right way to go about file locking in *nix.. the second is a script i found in ASPNs Python Cookbook which can lock files on Win32 and *nix:

    http://www.python.org/doc/current/lib/module-fcntl.html
    http://aspn.activestate.com/ASPN/Coo...n/Recipe/65203

    Mark.
    programming language development: www.netytan.com Hula

  12. #7
  13. No Profile Picture
    Junior Member
    Devshed Newbie (0 - 499 posts)

    Join Date
    Oct 2003
    Location
    CA
    Posts
    5
    Rep Power
    0
    Thank you very much! This really helps.

IMN logo majestic logo threadwatch logo seochat tools logo