How does one use both .tp_getattro / .tp_setattro and .tp_getset in a custom PyTypeObject?

Question

I'm working to extend an existing C application into Python that includes custom data types. I would like the user to have the capability to pass an arbitrary string as an attribute for objects of this custom type. Some special attributes would always be present and have a known behavior (e.g. CustomObject()._ndim). Any other string may or may not exist in the object and requires additional processing (e.g. CustomObject().asdf).

To solve this and stay as close to the standard API as possible, I would like to take advantage of the benefits of both the .tp_getattro / .tp_setattro and the .tp_getset features of the PyObjectType struct. .tp_getset provides a clean interface for defining getters and setters, documentation, and automatically end up in dir(). .tp_getattro / .tp_setattro are necessary to provide the dynamic functionality.

However, if both are provided as a PyTypeObject instance, the resulting type ignores the .tp_getset functions because .tp_getattro / .tp_setattro have been overridden. I assume there is a call or short stanza I need to add to my .tp_getattro / .tp_setattro implementations to make this work.

So far I've tried walking back through GDB the backtrace when my *attro() overrides are called. I see there's a split in the call stack depending on which is implemented, suggesting PyObject_GenericGetAttr() is involved.

.tp_getset Implemented

[0] from 0x00007ffff74700d6 in Info_get_ndim+17 at info.c:396
[1] from 0x00000000004c4419 in getset_get+185 at ../Objects/typeobject.c:1399
[2] from 0x000000000059b20f in _PyObject_GenericGetAttrWithDict+730 at ../Objects/typeobject.c:3125
[3] from 0x000000000059b20f in PyObject_GenericGetAttr+730 at ../Objects/object.c:1306
[4] from 0x000000000059b20f in PyObject_GetAttr+799 at ../Objects/object.c:912
[5] from 0x0000000000541fb5 in _PyEval_EvalFrameDefault+1077 at ../Python/ceval.c:2573
# ...

.tp_getattro Implemented

[0] from 0x00007ffff74700d6 in Info_getattro+17 at info.c:185
[1] from 0x000000000059b20f in PyObject_GetAttr+434 at ../Objects/dictobject.c:1333
[2] from 0x0000000000541fb5 in _PyEval_EvalFrameDefault+1077 at ../Python/ceval.c:2573
# ...

How does one use both .tp_getattro / .tp_setattro and .tp_getset in a custom PyTypeObject?

Thank you!

Answer 1

Thanks to @DavidW in the comments to the question, I've come across a solution: the .tp_getattro / .tp_setattro functions must call PyObject_GenericGetAttr() / PyObject_GenericSetAttr() respectively.

It makes sense why this wouldn't happen automatically: now you have more control. You can control which attributes you want to take precedence, what to do if an attribute doesn't exist. In my case, I want the dynamic properties to have precedence, and I don't want the user to be able to add their own attributes. So my *attro()s look similar to this:

Custom_getattro()

static PyObject * Custom_getattro(PyObject * o, PyObject * attr_name) {
  CustomObject * s = (CustomObject *) o;
  PyObject * res, * type, * value, * traceback;

  // First try to get a dynamic attribute
  res = _get_custom(s, attr_name);

  // If that was unsuccessful, get an attribute out of our .tp_dict
  if (! res) {
    PyErr_Fetch(&type, &value, &traceback);
    res = PyObject_GenericGetAttr(o, attr_name);

    // Use the original error, if necessary
    if (! res) {
      PyErr_Restore(type, value, traceback);
    } else {
      Py_XDECREF(type);
      Py_XDECREF(value);
      Py_XDECREF(traceback);
    }
  }

  return res;
}

Custom_setattro()

static int32_t Custom_setattro(PyObject * o, PyObject * attr_name, PyObject * v) {
  int32_t rc = -1;
  CustomObject * s;
  PyObject * type, * value, * traceback;

  // First try to set a dynamic attribute.
  rc = _set_custom(o, attr_name);

  // If that was unsuccessful, set an attribute in our .tp_dict
  if (rc != 0) {
    PyErr_Fetch(&type, &value, &traceback);
    rc = PyObject_GenericSetAttr(o, attr_name, v);

    // Use the original error, if necessary
    if (rc != 0) {
      PyErr_Restore(type, value, traceback);
    } else {
      Py_XDECREF(type);
      Py_XDECREF(value);
      Py_XDECREF(traceback);
    }
  }

  return rc;
}