Search code examples
pythoncc-preprocessorpython-cffi

How do I preprocess a header file with macros to a non macro header file


C newbie here. My use case is to give a header file posted here to the python library cffi so that I can bind to a C library. The header file in the previous link has macros. cffi only accepts a header file without macros, I think.

  • Is there a way to give a header file with macros to cffi by specifying any options?
  • If not, how do I preprocess that file so that I can give it to cffi? I tried gcc -E ddlog.h > ddlog-processed.h but when I ran the following code with cffi, it errors out.

Steps to reproduce:

  • Download ddlog.h
  • pip install cffi
  • gcc -E ddlog.h > ddlog-processed.h
  • In a file build.py in the same folder as ddlog.h, place
import cffi
import pathlib

ffi = cffi.FFI()
this_dir = pathlib.Path().absolute()
h_file_name = this_dir / "ddlog-processed.h"
with open(h_file_name) as h_file:
    ffi.cdef(h_file.read())

  • python build.py

It gives the error:

Traceback (most recent call last):
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/cparser.py", line 305, in _parse
    ast = _get_parser().parse(fullcsource)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/pycparser/c_parser.py", line 152, in parse
    debug=debuglevel)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 331, in parse
    return self.parseopt_notrack(input, lexer, debug, tracking, tokenfunc)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 1199, in parseopt_notrack
    tok = call_errorfunc(self.errorfunc, errtoken, self)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/pycparser/ply/yacc.py", line 193, in call_errorfunc
    r = errorfunc(token)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/pycparser/c_parser.py", line 1861, in p_error
    column=self.clex.find_tok_column(p)))
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/pycparser/plyparser.py", line 67, in _parse_error
    raise ParseError("%s: %s" % (coord, msg))
pycparser.plyparser.ParseError: /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/sys/_types/_int8_t.h:30:18: before: char

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/bin/invoke", line 10, in <module>
    sys.exit(program.run())
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/invoke/program.py", line 384, in run
    self.execute()
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/invoke/program.py", line 566, in execute
    executor.execute(*self.tasks)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/invoke/executor.py", line 129, in execute
    result = call.task(*args, **call.kwargs)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/invoke/tasks.py", line 127, in __call__
    result = self.body(*args, **kwargs)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/tasks.py", line 48, in build_cffi
    ffi.cdef(h_file.read())
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/api.py", line 112, in cdef
    self._cdef(csource, override=override, packed=packed, pack=pack)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/api.py", line 126, in _cdef
    self._parser.parse(csource, override=override, **options)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/cparser.py", line 358, in parse
    self._internal_parse(csource)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/cparser.py", line 363, in _internal_parse
    ast, macros, csource = self._parse(csource)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/cparser.py", line 307, in _parse
    self.convert_pycparser_error(e, csource)
  File "/Users/rabraham/Documents/dev/ddlog/playpen_ddlog/venv/lib/python3.7/site-packages/cffi/cparser.py", line 336, in convert_pycparser_error
    raise CDefError(msg)
cffi.CDefError: parse error
/Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/usr/include/sys/_types/_int8_t.h:30:18: before: char

Solution

  • There are two steps to using CFFI. The first one is to decide if you're using the ABI or the API mode---ffi.dlopen() versus ffi.set_source(); see https://cffi.readthedocs.io/en/latest/overview.html#abi-versus-api for details. Then you need to write the call to ffi.cdef() accordingly. I generally recommend to use the API mode, which is much more flexible at the cost of requiring a C compiler at install-time (just like if you wrote a standard CPython C extension module). In both modes, you should copy manually parts of the C header inside the call to ffi.cdef(); in the API mode you can leave many more details out and replace them with ... (dot-dot-dot).

    In neither of the two modes can you just paste a random C header making use of all the standard C features. Calling gcc -E just makes the problem even harder (but has been done in some cases with very large libraries, with lots of custom post-processing). The point of ffi.cdef() is that normally, you paste a simplified version of just the features you need.

    To answer your precise question:

    • macros that are just constants, like an integer, are directly supported. In API mode you can also write #define MY_CONSTANT ... with dot-dot-dot.

    • macros that work like functions are supported in the API mode (only) by using the following trick: write them in ffi.cdef() as if they were regular functions. That's enough to be able to call them.

    • for more complex cases, you might have to write a real C function that wraps the usage of the macro or macros as needed. Just invent a new function name, declare the function in ffi.cdef(), and finally implement the function directly in ffi.set_source().

    In the ABI mode, the last two cases cannot be supported at all---macros don't exist in the ABI any more.