How to access sys.stderr after calling PyErr_Print with python c-api

Question

I am trying to use the python c-api from a c# project via dll import.

I am getting an ModuleNotFoundError when importing some modules, which I thought are builtin. (Note: I compiled python myself)

I am a bit stuck right now, but my hope is to get some extra information when calling PyErr_Print() in the code below.

Code:

IntPtr modulePtr = NativeInterface.PyImport_ExecCodeModuleEx(moduleName,compiledModule, path);
if (modulePtr == IntPtr.Zero)
{
  NativeInterface.PyErr_Print();
  PythonException exception = PythonException.Query();
  throw exception;
}

The docs for PyErr_Print state that it will populate sys.stderr with some error information. What would be the easiest way to read this variable from my c# application?

Answer 1

This answer gives C code because I understand C and not C#, but I think it should be pretty transferable.

By default sys.stderr writes to some console somewhere and so you can't meaningfully try to read from it. However, it's perfectly possible to replace it to redirect the output. Two sensible options include writing to a file, and writing to a StringIO object that can later be queried.

The C code to run is basically equivalent to:

import sys
from io import StringIO # Python 3
sys.stderr = StringIO()

or in C:

int setup_stderr() {
    PyObject *io = NULL, *stringio = NULL, *stringioinstance = NULL;

    int success = 0;

    io = PyImport_ImportModule("io");
    if (!io) goto done;
    stringio = PyObject_GetAttrString(io,"StringIO");
    if (!stringio) goto done;
    stringioinstance = PyObject_CallFunctionObjArgs(stringio,NULL);
    if (!stringioinstance) goto done;

    if (PySys_SetObject("stderr",stringioinstance)==-1) goto done;

    success = 1;

    done:
    Py_XDECREF(stringioinstance);
    Py_XDECREF(stringio);
    Py_XDECREF(io);
    return success;
}

You run this once at the start of your program.

To query the contents of sys.stderr you'd then do the equivalent of:

value = sys.stderr.getvalue()
encoded_value = value.encode() # or you could just handle the unicode output

In C:

char* get_stderr_text() {
    PyObject* stderr = PySys_GetObject("stderr"); // borrowed reference

    PyObject *value = NULL, *encoded = NULL;

    char* result = NULL;
    char* temp_result = NULL;
    Py_ssize_t size = 0;

    value =  PyObject_CallMethod(stderr,"getvalue",NULL);
    if (!value) goto done;
    // ideally should get the preferred encoding
    encoded = PyUnicode_AsEncodedString(value,"utf-8","strict");
    if (!encoded) goto done;
    if (PyBytes_AsStringAndSize(encoded,&temp_result,&size) == -1) goto done;
    size += 1;

    // copy so we own the memory
    result = malloc(sizeof(char)*size);
    for (int i = 0; i<size; ++i) {
        result[i] = temp_result[i];
    }

    done:
    Py_XDECREF(encoded);
    Py_XDECREF(value);

    return result;

}

There's a bit of effort spent copying the string. You might consider working directly with unicode and using PyUnicode_AsUCS4Copy.

You probably then want to look at clearing the string after you've written it, just done by replacing sys.stderr with a fresh StringIO object.