Structure JSON to an `attrs` class with extra fields using `cattrs`?

Question

I would like to structure JSON to an attrs class that allows for extra fields using cattrs. cattrs by default will ignore extra fields and if forbid_extra_keys=True an error is raised when extra fields are passed.

I would like to do kind of the opposite: change the default behavior by allowing extra fields. I created an attrs class to do so but I'm a bit unsure on how to proceed with the custom cattrs converter. Here's what I have so far:

import attr
from cattr.preconf.json import make_converter


@attr.s(auto_detect=True)
class ClassWithExtras:
    foo: int

    def __init__(self, **attributes) -> None:
        for field, value in attributes.items():
            if field in self.__attrs_attrs__:
                self.__attrs_init__(field=value)
            else:
                setattr(self, field, value)

converter = make_converter()
converter.register_structure_hook_func(
    lambda cls: issubclass(cls, ClassWithExtras), lambda attribs, cls: cls(**attribs)
)

structured = converter.structure({"foo": "2", "bar": 5}, ClassWithExtras)

My issue is that since we are basically just unpacking the dictionary in the class, the types are not correct. I.e.: doing something like structured.foo + structured.bar will raise an error that we cannot concatenate/sum str and int.

Is there a way to do this in cattrs/attrs?

Answer 1

What you're trying to do is a little inadvised; the entire point of attrs classes is for all the fields to be enumerated in advance. If you stick arbitrary attributes on instances, you have to use non-slot classes, your helper functions like __repr__ and __eq__ won't work properly (the extra attributes will be ignored), and as you correctly concluded cattrs cannot help you with type conversions (since it has nowhere to actually find the types).

That said, I have rewritten your example to move the logic from the class into a converter, which I find more elegant.

from typing import Any
from attr import define, fields

from cattr.gen import make_dict_structure_fn
from cattr.preconf.json import make_converter


@define(slots=False)
class ClassWithExtras:
    foo: int


converter = make_converter()


def make_structure(cl):
    # First we generate what cattrs would have used by default.
    default_structure = make_dict_structure_fn(cl, converter)

    # We generate a set of known attribute names to use later.
    attribute_names = {a.name for a in fields(cl)}

    # Now we wrap this in a function of our own making.
    def structure(val: dict[str, Any], _):
        res = default_structure(val)
        # `res` is an instance of `cl` now, so we just stick
        # the missing attributes on it now.
        for k in val.keys() - attribute_names:
            setattr(res, k, val[k])
        return res

    return structure


converter.register_structure_hook_factory(
    lambda cls: issubclass(cls, ClassWithExtras), make_structure
)

structured = converter.structure({"foo": "2", "bar": 5}, ClassWithExtras)
assert structured.foo == 2
assert structured.bar == 5

This essentially does what your example does, just using cattrs instead of attrs.

Now, I also have a counter proposal. Let's say instead of sticking the extra attributes directly on the class, we gather them up into a dictionary and stick that dictionary into a regular field. Here's the entire example, rewritten:

from typing import Any

from attr import define, fields

from cattr.gen import make_dict_structure_fn
from cattr.preconf.json import make_converter


@define
class ClassWithExtras:
    foo: int
    extras: dict[str, Any]


converter = make_converter()


def make_structure(cl):
    # First we generate what cattrs would have used by default.
    default_structure = make_dict_structure_fn(cl, converter)

    # We generate a set of known attribute names to use later.
    attribute_names = {a.name for a in fields(cl)}

    # Now we wrap this in a function of our own making.
    def structure(val: dict[str, Any], _):
        val["extras"] = {k: val[k] for k in val.keys() - attribute_names}
        res = default_structure(val)
        return res

    return structure


converter.register_structure_hook_factory(
    lambda cls: issubclass(cls, ClassWithExtras), make_structure
)

structured = converter.structure({"foo": "2", "bar": 5}, ClassWithExtras)
assert structured.foo == 2
assert structured.extras["bar"] == 5

assert structured == ClassWithExtras(2, {"bar": 5})