Search code examples
multithreadingrustconcurrencymultiprocessingthread-safety

is there an equivalent of Python's multiprocessing module for Rust


In python there's a multiprocessing module where you can run a function inside a child process, I know what you're thinking, but Rust doesn't have Gil, wouldn't threads do the same thing for you?

No, because I have tasks that connect to different databases at the same time using different odbc drivers, a lot of the odbc drivers are not thread safe, (I don't know if it's thread safety but if I run 2 different drivers at the same time in different threads, that doesn't work for some reason)

please don't focus on the ODBC drivers part, it's a long story and it's specific to ODBC, I'm sure others might need to run Rust functions in a process instead of threads for various reasons, most of them won't be valid but some of them would and that's a bit besides the point of this question

so I need to run each parallels task in separate memory scope, aka a process.

I don't mind the complexity involving passing values and variable between processes and child\parent process, I think I know how to do that (at least how to do it using Redis)

in python this is done using this:

from multiprocessing import Process

    def f(name):
        print('hello', name)
    
    if __name__ == '__main__':
        p = Process(target=f, args=('bob',))
        p.start()
        p.join()

notice how the parameter "target" in "Process" takes a function.

I know Rust have Process too but it doesn't take a Rust function to run in a separate process, it takes a binary, that might work but it's harder to develop, I have to compile a separate rust project as a binary for this to work, while if it can just run a function\closure in a child process, then it could be one project

what I'm looking for is something like this, but instead of "tokio" imagine it's "process" (so process::spawn)

#[tokio::main]
async fn main() -> io::Result<()> {
        // replace tokio::spawn with process::spawn
        tokio::spawn(async move {
            dostuff.await
        });
}

Solution

  • As a matter of fact, there is: the crate mitosis offers a primitive analogous to thread::spawn, which spawns a subprocess instead a thread. However, the mechanism by which it does so is pretty sketchy, full of loopholes, shaky assumptions and introducing a number of security weaknesses. You should consider it a proof-of-concept experiment more than a practical implementation.

    The vast majority of the time, you don’t need to have a separate process. Any issues with non-thread-safety and non-reentrancy should be possible to mitigate by not accessing any problematic objects outside a single dedicated thread, and using thread-safe primitives to have it intermediate between your threaded code and non-thread-safe code.