Error in writing Python functions in Crystal-lang

Question

I am trying to write some python function in crystal-lang through the C Python API.

My code follows:

METH_VARARGS  = 0x0001

@[Link("python3.5m")]
lib Python
  alias PyObject = Void*

  struct PyMethodDef
    name  : UInt8*
    func  : Void*
    flags : LibC::Int
    doc   : UInt8*
  end

  fun Py_Initialize
  fun Py_Finalize
  fun PyObject_CallObject(func : PyObject, args : PyObject) : PyObject
  fun PyCFunction_NewEx(method : PyMethodDef*, __self__ : PyObject, ) : PyObject
  fun PyLong_AsLong(n : PyObject) : Int64
  fun PyLong_FromLong(n : Int64) : PyObject

end

def new_method_def(name : String, function, flags : LibC::Int)
  x = Pointer(Python::PyMethodDef).malloc(1)
  x.value.name  = name.to_unsafe
  x.value.func  = function
  x.value.flags = flags
  x.value.doc   = nil
  x
end

Python.Py_Initialize

a = ->(args : Void*) { 
                       puts Python.PyLong_AsLong(args)
                       Pointer(Void).null 
                     }

name     = "num"
number   = Python.PyLong_FromLong(1) 
Python.Py_IncRef(number)
method   = Python.PyCFunction_NewEx(new_method_def(name,a.pointer,METH_VARARGS),number)
Python.PyObject_CallObject(method,Pointer(Void).null)

Python.Py_Finalize

Everything works if I set nil instead of number when in PyCFunction_NewEx, but as the code is, it throws an invalid acces memory exception when Py_Finalize is called. I can't understand what's causing it. Can someone help me?

Answer 1

The root problem here is that you're calling a C function of three parameters with only two arguments.

Regrettably, PyCFunction_NewEx is missing from the documentation, despite being a public API function. But all of the examples using it pass three arguments. And if you go to the source:

PyObject *
PyCFunction_NewEx(PyMethodDef *ml, PyObject *self, PyObject *module)

That's 3.7, but this is the same in 3.5 and in 2.7, and in every other version since the function was added to the API in 2.3. The whole point of NewEx is to allow you to pass a module.

Presumably, the function is expecting that third argument either in a register or on the stack, and you haven't put anything there, so it's completely arbitrary what you're passing. Slightly different code will leave completely different values in those places, so it's not surprising that you get different results:

• If the value happens to be 0, that's fine; you're allowed to pass NULL as the module value.
• If the value happens to be something that points to unmapped memory, like, say, 1 (as in the raw C long/long long, not a PyLongObject), you should get a segfault from the attempt to incref the module.
• If the value happens to be a pointer to some random thing in memory, the incref will work, but will corrupt that random thing. Which could do just about anything, but a mysterious segfault at some arbitrary later point is almost the least surprising thing it could do.

---

Meanwhile, from a comment:

I am calling PyCFunction_NewEx because PyCFunction_New is a marco in the source code.

If you're using Python 2.3-2.6 or 3.0-3.2, then sure. But in later versions, including the 3.5 you say you're using, CPython goes out of its way to define PyCFunction_New as a function specifically so that it will be present in the API (and even the stable API, for 3.x). See 3.5 for example:

/* undefine macro trampoline to PyCFunction_NewEx */
#undef PyCFunction_New

PyAPI_FUNC(PyObject *)
PyCFunction_New(PyMethodDef *ml, PyObject *self)
{
    return PyCFunction_NewEx(ml, self, NULL);
}

So, you really can just call PyCFunction_New.