Search code examples
pythonc++multiprocessingpython-multiprocessingboost-python

Is it possible to force multiple python processes to share the same memory via a C++ extension?


Here is my attempt:

int* globalvar = new int[8];
void cpp_init(){
    for (int i = 0; i < 8; i++)
        globalvar[i] = 0;
}
void writeAtIndex(int index, int value){
    globalvar[index] = value;
}
int accessIndex(int index){
    return globalvar[index];
}
BOOST_PYTHON_MODULE(MpUtils){
    def("cpp_init", &cpp_init);
    def("writeAtIndex", &writeAtIndex);
    def("accessIndex", &accessIndex);
}

and in the python file

def do_stuff(index):
    writeAtIndex(index, randint(1, 100))
    time.sleep(index/10)
    print([accessIndex(i) for i in range(8)])

if __name__ == '__main__':
    freeze_support()
    processes = []
    cpp_init()
    for i in range(0, 10):
        processes.append( Process( target=do_stuff, args=(i,) ) )
    for process in processes:
        process.start()
    for process in processes:
        process.join()

And the output is this:

[48, 0, 0, 0, 0, 0, 0, 0]
[0, 23, 0, 0, 0, 0, 0, 0]
[0, 0, 88, 0, 0, 0, 0, 0]
[0, 0, 0, 9, 0, 0, 0, 0]
[0, 0, 0, 0, 18, 0, 0, 0]
[0, 0, 0, 0, 0, 59, 0, 0]
[0, 0, 0, 0, 0, 0, 12, 0]
[0, 0, 0, 0, 0, 0, 0, 26]

Could someone explain why this is not working? I tried printing the globalvar and it's always the same value. There shouldn't be any race conditions as wating for 0.1 to 0.8 seconds should be more than enough to make the computer write something. Shouldn't C++ be accessing directly the location of the pointer?

Thanks


Solution

  • Processes can generally access only their own memory space. You can possibly use the shared_memory module of multiprocessing to share the same array across processes. See example in the linked page.