Python environ variables strange behaviour when doing multiprocessing

Question

I was struggling with some strange behaviour in Python 3.9 when using multiprocessing and environment variables.

I have created the following example that reproduces the issue:

import multiprocessing.pool
import os
import copy
import time

multiprocessing.set_start_method('fork')
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
WORKERS = 2


def method(index: int):
    print(f'* index: {index}; position: 1; env var value: {os.environ["CUDA_VISIBLE_DEVICES"]} *')
    os.environ["CUDA_VISIBLE_DEVICES"] = str(index)
    print(f'* index: {index}; position: 2; env var value: {os.environ["CUDA_VISIBLE_DEVICES"]} *')
    time.sleep(3)  # some task
    print(f'* index: {index}; position: 3; env var value: {os.environ["CUDA_VISIBLE_DEVICES"]} *')


pool = multiprocessing.pool.ThreadPool(processes=WORKERS)
pool.map(lambda index: method(index), list(range(WORKERS)))

The idea is to have a pool of two threads with a distinct value for the CUDA_VISIBLE_DEVICES environment variable.

The expected output would be something like the following:

* index: 0; position: 1; env var value: 0 *
* index: 0; position: 2; env var value: 0 *
* index: 1; position: 1; env var value: 0 *
* index: 1; position: 2; env var value: 1 *
* index: 1; position: 3; env var value: 1 *
* index: 0; position: 3; env var value: 0 *

The thread 0 has always value 0 for the env variable, whereas the thread 1 changes to value 1 after the 1st position.

However the resulting output is the following:

* index: 0; position: 1; env var value: 0 *
* index: 0; position: 2; env var value: 0 *
* index: 1; position: 1; env var value: 0 *
* index: 1; position: 2; env var value: 1 *
* index: 1; position: 3; env var value: 1 *
* index: 0; position: 3; env var value: 1 *

In this case, the thread 0 changes its value to 1 in the 3rd position, after the time sleep.

Why is this happening? Is this something expected with environment variables? Maybe os.environ has some shared nature? I have tried to include the line os.environ = copy.deepcopy(os.environ.copy()) at the beginning of the function method, but it did not change the output. Neither including the following lines at the beginning worked:

os.environ = {}
os.environ["CUDA_VISIBLE_DEVICES"] = "0"

Is there a way to make this not happening?

Answer 1

As commented by @user2357112, the issue was using a multiprocessing.pool.ThreadPool instead of a multiprocessing.pool.Pool. Threads share the environment, whereas processes do not. Aside from this, threads do not surpass the GIL lock of Python, not doing a proper parallelism.