Search code examples
pythonmysqlclassblobcherrypy

reconstituting a class full of data from MySQL BLOB object in python

I'm using CherryPy and it seems to not behave nicely when it comes to retrieving data from stored files on the server. (I asked for help on that and nobody replied, so I'm on to plan B, or C...) Now I have stored a class containing a bunch of data structures (3 dictionaries and two lists of lists all related) in a MySQL table, and amazingly, it was easier than I thought to insert the binary object (longblob). I turned it into a pickle file and INSERTED it.

However, I can't figure out how to reconstitute the pickle and rebuild the class full of data from it now. The database returns a giant string that looks like the pickle, but how to you make a string into a file-like object so that pickle.load(data) will work?

Alternative solutions: How to save the class as a BLOB in database, or some ideas on why I can save a pickle of this class but when I go to load it later, the class seems to be lost. But in SSH / locally, it works - only when calling pickle.load(xxx) from cherrypy do I get errors.

I'm up for plan D - if there's a better way to store a collection of structured data for fast retrieval without pickles or MYSQL blobs...

Solution

  • You can create a file-like in-memory object with (c)StringIO:

    >>> from cStringIO import StringIO
>>> fobj = StringIO('file\ncontent')
>>> for line in fobj:
...   print line
...
file

content

    But for pickle usage you can directly load and dump to a string (have a look at the s in the function names):

    >>> import pickle
>>> obj = 1
>>> serialized = pickle.dumps(obj)
>>> serialized
'I1\n.'
>>> pickle.loads(serialized)
1

    But for structured data stored in a database, I would suggest in general that you either use

    • a table, preferable with an ORM like sqlalchemy so it is directly mapped to a class or
    • a dictionary, which could be easily (de)serialized with JSON

    and not using pickle at all.