Embedded Python (C API): How to run .py file repeatedly without reinitializing

Question

I'm having a heck of a hard time with embedded python.

I'm running a DLL, and each time a certain function is called I want to run a Python script. I want to avoid calling Py_Finalize() at the end of the function because the Initialize/Finalize calls account for about 75ms of time, and I can't afford that.

Problem is, I can't seem to run the same .py file multiple times without errors.

...
runResult = PyRun_SimpleFile(pyfileptr, pyfilename);
if (runResult)
{
    if (!PyErr_Occurred())
        return -4;
    PyErr_Print();
    return -3;
}
...

I always end up returning -4 the second time through. I don't even know how that's possible, because the documentation says PyRun_SimpleFile returns -1 if there was an exception and 0 otherwise, but PyErr_Occurred() returns NULL for no exception.

Even when the Python file I'm running is simply

print("hi")

I end up with the same results, which certainly leads me to believe it's not an exception generated by the script itself.

UPDATE: It's looking more and more like this is a DLL-related issue, as running the same code in a standalone application doesn't show the problem. Still pretty stumped though.

Answer 1

OP here. I'd basically posed two questions, which I have somewhat poor answers to now:

How do I run a python file without reinitializing? Just don't call finalize before calling PyRun_SimpleFile() or boost::python::exec_file() a second time.

Why is PyErr_Occurred() returning 0 after PyRun_SimpleFile() returns non-zero? The short answer is I still don't know, but my best guess is that it is related to the DLL implementation and some hanging or missing reference.

I used boost::python based on kichik's suggestion, and while I wouldn't say it's a lot easier to use than the base C API, it is easier to read. It also did not exhibit the missing error problem, so ultimately it solved my problem. I was able to do two consecutive exec_file() calls without a problem, even in a DLL.

Because I had some trouble finding examples of boost::python used in the way I needed to, I'll put my code here, slightly trimmed for space. Of course some of this is specific to my project but it may still be valuable as a general example.

extern "C" LTPYTHON_API int ltPythonAnalyzeLog(char * analyzerfile, char * logfile, double timeWindow, int * results)
{

std::vector<int> countsVector;
Py_Initialize();
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
    // Example of adding a variable to the global namespace
main_namespace["scriptIsRunningThroughDll"] = boost::python::long_(1);

// Load arguments for the analyzer call
{
    int argc = 3;
    wchar_t * argv[3];

    //*... assemble wchar arguments for py script ... *

    PySys_SetArgv(argc, argv);
}

int startClock = clock();
try
{
    exec_file(analyzerfile, main_namespace);
}
catch(error_already_set const &)
{
            //*... some error handling ...*

    PyObject *ptype, *pvalue, *ptraceback;
    PyErr_Fetch(&ptype, &pvalue, &ptraceback);

    handle<> hType(ptype);
    object extype(hType);
    handle<> hTraceback(ptraceback);
    object traceback(hTraceback);

    //Extract error message
    std::string strErrorMessage = extract<std::string>(pvalue);
    long lineno = extract<long> (traceback.attr("tb_lineno"));

    FILE * outfile = fopen("ltpython-error.txt", "a");
    fprintf(outfile, "%d: %s\n", lineno, strErrorMessage);
    fflush(outfile);
    fclose(outfile);

    return -1;
}

    //*... grabbing a matrix of results that were created in the script ...*
object counts = main_namespace["sortedIndicationCounts"];
list countsList = extract<list>(counts);
int totalCount = 0;
for (int i = 0; i < len(countsList); i++)
{
    list singleCount = extract<list>(countsList[i]);
    countsVector.push_back(extract<int>(singleCount[1]));
    totalCount += countsVector[i];
}

    //*... returning the number of milliseconds that elapsed ...*
return clock() - startClock;
}

The error handling is based on this answer.