Can two concurrent Python processes run in Boost Python?

Question

I am trying to run concurrent Python scripts in a Python interpreter from boost::python. Here is a short sample program:

#include <Python.h>

#include <boost/python/exec.hpp>
#include <iostream>
#include <thread>

#include <boost/python/extract.hpp>
#include <boost/python/import.hpp>
#include <boost/python/object.hpp>

int main(int argc, char const *argv[]) {

  const char *prog = "def ack(m, n):\n"
                     "  if m == 0:\n"
                     "    return n + 1\n"
                     "  elif n == 0:\n"
                     "    return ack(m - 1, 1)\n"
                     "  else:\n"
                     "    return ack(m - 1, ack(m, n - 1))";

  Py_Initialize();
  try {
    std::thread t1([&prog]() {
      std::cout << "t1" << std::endl;

      boost::python::object mainModule = boost::python::import("__main__");
      boost::python::object mainNamespace = mainModule.attr("__dict__");

      boost::python::exec(prog, mainNamespace, mainNamespace);
      int val = boost::python::extract<int>(
          boost::python::eval("ack(3,3)", mainNamespace, mainNamespace));
      std::cout << "t1 result: " << val << std::endl;
    });

    std::thread t2([&prog]() {
      std::cout << "t2" << std::endl;

      boost::python::object mainModule = boost::python::import("__main__");
      boost::python::object mainNamespace = mainModule.attr("__dict__");

      boost::python::exec(prog, mainNamespace, mainNamespace);
      int val = boost::python::extract<int>(
          boost::python::eval("ack(3,4)", mainNamespace, mainNamespace));
      std::cout << "t2 result: " << val << std::endl;
    });

    t1.join();
    t2.join();
  } catch (boost::python::error_already_set const &e) {
    PyErr_Print();
  }

  return 0;
}

The problem is that the program fails intermittantly. I've tried it on two different Linux boxes. On one, it fails about 3/4 of the time; on the other about 1/10 of the time. The most common failure mesaage is the less than helpful:

RUN FINISHED; Segmentation fault; core dumped; real time: 60ms; user: 0ms; system: 0ms

There are several tempting calls in the Boost Pythnon documentation refering to concurrancy, but none of the combinations I've tried help. The global interpreter lock (GIL) seems to be desinged to allow C++ to access the same Python interpreter thread, unless I'm misunderstanding it. I want two completely independent Python interpreters running at the same time.

This example is close, but uses two separate C threads that occasionally call Python to do some work. I'm trying to run two separate Python processes concurrently.

This question is similar, although it gives less detail. In my case, the Python interpreters are used to tie together external processes that will take considerable time running on supercomputers; they do not need to call back into my C++ app. Once they complete, the results are collected and displayed by the C++ app from external dropping files.

Answer 1

AFAIK, John is corrent. We never did find a way to run two concurrent Python projects in Boost Python. There are three ways to dodge this issue. Here's the first: Run two different Python interpreters.

#include <Python.h>

#include <boost/python/exec.hpp>
#include <iostream>
#include <thread>
#include <sys/wait.h>

#include <boost/python/extract.hpp>
#include <boost/python/import.hpp>
#include <boost/python/object.hpp>

void python (std::string fork, int m, int n) {
  const char *prog = "def ack(m, n):\n"
                     "  if m == 0:\n"
                     "    return n + 1\n"
                     "  elif n == 0:\n"
                     "    return ack(m - 1, 1)\n"
                     "  else:\n"
                     "    return ack(m - 1, ack(m, n - 1))";

  Py_Initialize();
  try {
    std::cout << fork << std::endl;

    boost::python::object mainModule = boost::python::import("__main__");
    boost::python::object mainNamespace = mainModule.attr("__dict__");

    std::stringstream commandstream;
    commandstream << "ack(" << m << "," << n << ")";
    std::string command = commandstream.str();
    boost::python::exec(prog, mainNamespace, mainNamespace);
    int val = boost::python::extract<int>(boost::python::eval(command.c_str(), mainNamespace, mainNamespace));
    std::cout << fork << " result: " << val << std::endl;
  } catch (boost::python::error_already_set const &e) {
    PyErr_Print();
  }
}

int main (int argc, char const *argv[]) {
  pid_t pid = fork();
  if (pid == 0) {
    python("f1", 3, 4);
  } else if (pid > 0) {
    python("f2", 3, 3);

    int status;
    waitpid(pid, &status, 0);
  } else {
    std::cout << "Fork failed." << std::endl;
  }

  return 0;
}

The second way, and the one we ended up using, is to place the code to run the Python interpreter in an external executable and run that.

The third is to block the thread until the first Python process finishes. That's viable if each Python process is expected to take a very short amount of time, but that was not the case in our application, so we rejected this alternative.