Can't pickle/dill SwigPyObject when serializing dict imoprted by importlib

Question

I try to serialize (dill) a list containing dill-able objects which is nested inside a dict. The dict itself is imported into my main script using importlib. Calling dill.dump() raises a TypeError: can't pickle SwigPyObject objects. Here is some code with which I managed to reproduce the error for more insight.

some_config.py located under config/some_config.py:

from tensorflow.keras.optimizers import SGD  
from app.feature_building import Feature

config = {
    "optimizer": SGD(lr=0.001),

    "features": [
        Feature('method', lambda v: v + 1)
    ],
}

Here is the code which imports the config and tries to dill config["features"]:

import dill
import importlib.util

from config.some_config import config

spec = importlib.util.spec_from_file_location(undillable.config,"config/some_config.py")
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
undillable_config = module.config

# Works prefectly fine
with open("dillable_config.pkl", "wb") as f:
    dill.dump(config["features"], f)

# Raises TypeError: can't pickle SwigPyObject objects
with open("undillable_config.pkl", "wb") as f:
    dill.dump(undillable_config["features"], f)

Now the part that made me wonder: When importing the config-dict with importlib it raises the error and after some debugging I found that not only config["features"] but also config["optimizer"] will be dilled. However, using normal import seems to work and it only tries to dill config["features"] So my question is why does dill try to serialize the whole dict if it is imported by importlib instead of only the feature-list and how may this error be fixed?

Answer 1

After reading the answer to this question I managed to get it working by avoiding importlib and instead import the config using __import__.

filename = "config/some_config.py"
dir_name = os.path.dirname(filename)
if dir_name not in sys.path:
    sys.path.append(dir_name)

file = os.path.splitext(os.path.basename(filename))[0]
config_module = __import__(file)

# Works prefectly fine now
with open("dillable_config.pkl", "wb") as f:
    dill.dump(config_module.config["features"], f)