I was recently experimenting with monkey patching built in types (Yes I know this is a terrible idea---trust me, this is for educational purposes only).
I discovered that there is an odd distinction between lambda expressions and functions declared with def. Take a look at this iPython session:
In [1]: %load_ext cython
In [2]: %%cython
...: from cpython.object cimport PyObject_GenericSetAttr
...: def feet_to_meters(feet):
...:
...: """Converts feet to meters"""
...:
...: return feet / 3.28084
...:
...: PyObject_GenericSetAttr(int, 'feet_to_meters', feet_to_meters)
In [3]: (20).feet_to_meters()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-3-63ba776af1c9> in <module>()
----> 1 (20).feet_to_meters()
TypeError: feet_to_meters() takes exactly one argument (0 given)
Now, if I wrap feet_to_meters with a lambda, everything works!
In [4]: %%cython
...: from cpython.object cimport PyObject_GenericSetAttr
...: def feet_to_meters(feet):
...:
...: """Converts feet to meters"""
...:
...: return feet / 3.28084
...:
...: PyObject_GenericSetAttr(int, 'feet_to_meters', lambda x: feet_to_meters(x))
In [5]: (20).feet_to_meters()
Out[5]: 6.095999804928006
What is going on with this?
Your problem can be reproduced in Python and without (very) dirty tricks:
class A:
pass
A.works = lambda x: abs(1)
A.dont = abs
A().works() # works
A().dont() # error
The difference is, that abs is a builtin-function of type PyCFunctionObject, while lambda is of type PyFunctionObject (a C is missing compared to PyCFunction...).
Those cfunctions just cannot be used for patching, see for example PEP-579.
The problem, which is also mentioned in PEP-579, that cython-functions are PyCFunctions and thus are seen as builtin-functions:
%%cython
def foo():
pass
>>> type(foo)
builtin_function_or_method
That means, you cannot use the Cython-function directly for monkey patching, but have to wrap them into a lambda or similar, as you already do. One should not worry about performance, because due to method-lookup there is already overhead, having a little bit more of it doesn't change things dramatically.
I must confess, I don't know why this is the case (historically). But in the current code (Python3.8) you can easily find the crucial line in _PyObject_GetMethod, which makes the difference:
descr = _PyType_Lookup(tp, name);
if (descr != NULL) {
Py_INCREF(descr);
if (PyFunction_Check(descr) || # HERE WE GO
(Py_TYPE(descr) == &PyMethodDescr_Type)) {
meth_found = 1;
} else {
After looking-up the function (here descr) in the dictionary _PyType_Lookup(tp, name), method_found is set to 1 only if the found function is of type PyFunction, which is not the case for builtin-PyCFunctions. Thus abs and Co aren't seen as methods, but stay kind of "staticmethod".
The easiest way to find a starting point for the investigation, is to inspect the produced opcode for:
import dis
def f():
a.fun()
dis.dis(f)
i.e. the following opcode(and seems to have changed since Python3.6):
2 0 LOAD_GLOBAL 0 (a)
2 LOAD_METHOD 1 (fun) #HERE WE GO
4 CALL_METHOD 0
6 POP_TOP
8 LOAD_CONST 0 (None)
10 RETURN_VALUE
We can inspect the corresponding part in ceval.c:
TARGET(LOAD_METHOD) {
/* Designed to work in tamdem with CALL_METHOD. */
PyObject *name = GETITEM(names, oparg);
PyObject *obj = TOP();
PyObject *meth = NULL;
int meth_found = _PyObject_GetMethod(obj, name, &meth);
....
and let the gdb take us from there.
As @user2357112 has rightly pointed out, if PyCFunctionObject would support the descriptor protocol (more precisely to provide tp_descr_get), even after meth_found = 0; it still would have a fall back which would lead to the desired behavior. PyFunctionObject does provide it, but PyCFunctionObject does not.
Older versions used LOAD_ATTR+CALL_FUNCTION for a.fun() and in order to work, function-objects had to support the descriptor protocol. But now it seems not to be mandatory.
My quick tests with extending the crucial line with PyCFunction_Check(descr) to:
if (PyFunction_Check(descr) || PyCFunction_Check(descr) ||
(Py_TYPE(descr) == &PyMethodDescr_Type))
have shown, that then also builtin-methods would work as bound-methods (at least for the case above). But this would probably break something - I didn't run any bigger tests.
However, as @user2357112 mentioned (thanks again), this would lead to a inconsistency, because meth = foo.bar still uses LOAD_ATTR and thus depends on the descriptor protocol.
Recommendation: I found this answer helpful in understanding, what is going on in the case of LOAD_ATTR.