Python C API, override the PyNumberMethods nb_add field also changes the field in tp_base

Question

I develop a new type: PyAlignArray_Type which derived from PyArray_Type

PyAlignArray_Type.tp_base = &PyArray_Type;

I override the nb_add field of tp_as_number as follow:

PyAlignArray_Type.tp_as_number->nb_add = (binaryfunc)ndarray_add ;

This works fine, but the field is also changed for base type (PyArray_Type). My question is: How to overriding a PyNumberMethods field without changing it in the base type ?

  typedef struct {
        PyArrayObject array;
        int  pitch;
        size_t size;
    } PyAlignArrayObject;

/*---------------PyAlignArray_Type------------------*/
static PyTypeObject PyAlignArray_Type = {
    PyObject_HEAD_INIT(NULL)
    "plibs_8.ndarray",  /* tp_name */
    sizeof(PyAlignArrayObject),      /* tp_basicsize */
    0,                       /* tp_itemsize */
    (destructor)array_dealloc,/* tp_dealloc */
    0,                       /* tp_print */
    0,                       /* tp_getattr */
    0,                       /* tp_setattr */
    0,                       /* tp_reserved */
    0,                       /* tp_repr */
    0,                       /* tp_as_number */
    0,                       /* tp_as_sequence */
    0,                       /* tp_as_mapping */
    0,                       /* tp_hash */
    0,                       /* tp_call */
    0,                       /* tp_str */
    0,                       /* tp_getattro */
    0,                       /* tp_setattro */
    0,                       /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT |
        Py_TPFLAGS_BASETYPE, /* tp_flags */
    0,                       /* tp_doc */
    0,                       /* tp_traverse */
    0,                       /* tp_clear */
    0,                       /* tp_richcompare */
    0,                       /* tp_weaklistoffset */
    0,                       /* tp_iter */
    0,                       /* tp_iternext */
    ndarray_methods,         /* tp_methods */
    ndarray_members,         /* tp_members */
    0,                       /* tp_getset */
    0,                       /* tp_base */
    0,                       /* tp_dict */
    0,                       /* tp_descr_get */
    0,                       /* tp_descr_set */
    0,                       /* tp_dictoffset */
    0,                       /* tp_init */
    0,                       /* tp_alloc */
    (newfunc)ndarray_new,    /* tp_new */

};




static PyModuleDef ndarraymodule = {
    PyModuleDef_HEAD_INIT,
    "ndarray",
    "ndarray module",
    -1,
    NULL, NULL, NULL, NULL, NULL
};

PyMODINIT_FUNC
PyInit_ndarray(void)
{
    PyObject *m;
    import_array();
    PyAlignArray_Type.tp_base = &PyArray_Type;
    if (PyType_Ready(&PyAlignArray_Type) < 0)
        return NULL;

    // __add__ overloading
    PyAlignArray_Type.tp_as_number->nb_add = (binaryfunc)ndarray_add ;

    m = PyModule_Create(&ndarraymodule);
    if (m == NULL)
        return NULL;

    Py_INCREF(&PyAlignArray_Type);
    PyModule_AddObject(m, "ndarray", (PyObject *) &PyAlignArray_Type);
    return m;
}

Answer 1

I think you're right - this is odd, non-intuitive behaviour. It is fixable though. The documentation says

The tp_as_number field is not inherited, but the contained fields are inherited individually.

which implies that it should be OK to override them as you've done. What it actually seems to mean is:

The tp_as_number field is not inherited, but the contained fields are inherited individually if you provide space for them. If you don't provide space for them then the tp_as_number field is set to point at the base classes tp_as_number field.

(Italics are my addition. See code at https://github.com/python/cpython/blob/3e8d6cb1892377394e4b11819c33fbac728ea9e0/Objects/typeobject.c#L5111 and https://github.com/python/cpython/blob/3e8d6cb1892377394e4b11819c33fbac728ea9e0/Objects/typeobject.c#L4757). This means that, as in your example, you're also changing the base class methods.

The solution to this is to create a zero initialized PyNumberMethods and reference that in your class

static PyNumberMethods align_array_number_methods = { NULL };
/* You could also specify the nb_add field here and you wouldn't
have to do it in your module init */

static PyTypeObject PyAlignArray_Type = {
    /* whole bunch of code unchanged */
    &align_array_number_methods,       /* tp_as_number */
    /* whole bunch of code unchanged */
    

The slots you left as NULL are filled in according to the inheritance rules in PyType_Ready

---

I think the reason for this odd behaviour is to avoid introducing memory leaks. When you call PyType_Ready (which is where the inheritance is dealt with) it can't allocate PyNumberMethods statically so it would have to to malloc it. Since you don't know whether it's malloced these fields or not the memory that never get freed. (This is only a concern for dynamically created types).