Search code examples
pythonclass-variablescloudpickle

Class variable is not shared over all instances after pickling and unpickling


I have a class ConfProvider that contains a class variable which holds all config that cannot be pickled. My other classes inherit from this class and they use the _conf class variable when they call the non-pickable objects. The idea is then to clean the config when pickling my classes so I do not get any errors because the class contains unpickable objects.

This is my ConfProvider class

class ConfProvider:
    _conf = None
    
    @staticmethod
    def set_conf(conf: dict) -> None:
        ConfProvider._conf = conf

    @staticmethod
    def clear_conf() -> None:
        ConfProvider._conf = {}

    @classmethod
    def print_conf(cls) -> None:
        cls._conf["console_logger"].info(ConfProvider._conf)

Next to this class I have 2 other classes that inherit from the ConfProvider. Class_1 contains objects of class_2.

class Class_1(ConfProvider):

    def __init__(self):
        # Do some init stuff
        self.list_class_2 = [Class_2(1, 2), Class_2(3, 4)]

    @Logging.debug_decorator
    def save(self, debug=False) -> None:
        config = self._conf
        self.clear_conf()

        with open(save_path, "wb") as f:
            cloudpickle.dump(self, f)

        # Reset non serializable attributes
        self.set_conf(config)

class Class_2(ConfProvider):

    def __init__(self, a, b):
        # Do some init stuff

I want to be able to pickle the Class_1 object I create that contains several objects of Class_2. For this, I use the save method as shown above in Class_1. Now this works perfectly since the class variable is shared amongst all objects. However, once I unpickle the class_1 object and add new class_2 objects to the list in class_1, the clear_conf method does only clear the conf variable from the objects that were included before pickling. The _conf of all new objects is not cleared. How is this possible as normally a class variable is shared over all instances?

EDIT:

After debugging a bit more, I found that before pickling Class_1._conf always contains the same content as Class_1().__class__._conf. However, once I pickle the class and unpickle it, the two are different. Class_1._conf has the values that were loaded when starting up, but Class_1().__class__._conf contains an empty dictionary.


Solution

  • Eventually I found the problem. I used the cloudpickle module, not the normal pickle module. Cloudpickle will pickle the objects by value instead of by reference. This way, the classes themselves were also pickled. When I unpickled my objects, the already loaded classes were overridden which caused that two ConfProvider classes were present, each with its own class variable.

    I was able to solve it by overriding the ConfProvider class that was loaded with the ConfProvider class present in the pickle file. Afterwards, everything worked as expected and the class variable was shared.