Python multiprocessing.Queue put kills process in Docker container

Question

I'm running a small multiprocessing test in a Docker container. The test uses multiprocessing.Queue with a consumer and a producer. The minimal code that leads to the crash can be found below, together with the Dockerfile. In the code, the main process starts a new process that receives a reference to the Queue, and uses Queue.put() to put an integer on the queue to the main process. What I'm seeing is that the producer process gets killed during the call to .put(), there is no Exception raised whatsoever.

Any ideas on why the .put() is killing the process? This does work locally (macOS) but does not work on the Python base image for a container. The Python version is 3.9.16. Update: this also does not work on a regular Debian VM.

import multiprocessing as mp
import time
import traceback
from typing import Any, Optional

import psutil

base_file = "logs.txt"


def main() -> None:
    queue: Any = mp.Queue()
    print("Queue created")
    print("Starting producer process")

    p = mp.get_context("spawn").Process(target=producer, args=(queue,), daemon=True)
    p.start()
    print(f"Main: producer started: {p.pid}")

    alive = True
    while alive:
        alive = p.is_alive()
        print(f"Ha ha ha staying alive, producer: {p.is_alive()}")
        time.sleep(1)

    print("Every process is dead :( ")

def producer(q: mp.Queue) -> None:
    with open(f"producer.{base_file}", "w") as f:
        print("Producer: started", file=f, flush=True)
        current_value: int = 0
        while True:
            print(f"Producer: Adding value {current_value} to queue", file=f, flush=True)
            try:
                q.put(current_value, block=False)
            except BaseException as e:
                print(f"Producer: exception: {e}", file=f, flush=True)
                print(f"{traceback.format_exc()}", file=f, flush=True)
                raise e
            print(f"Producer: Value {current_value} added to queue", file=f, flush=True)
            print("Producer: Sleeping for 1 second", file=f, flush=True)
            time.sleep(1)
            current_value += 1


if __name__ == "__main__":
    main()

FROM python:3.9.16

RUN apt-get update && apt-get install -y gettext git mime-support && apt-get clean

RUN python3 -m pip install psutil

COPY ./multiprocessing_e2e.py /src/multiprocessing_e2e.py

WORKDIR /src

CMD ["python", "-u", "multiprocessing_e2e.py"]

Answer 1

When trying to run your example in a fedora39, Python 3.12.4, WSL, I got an error message that likely explains the problem:

    raise RuntimeError('A SemLock created in a fork context is being '
RuntimeError: A SemLock created in a fork context is being shared with a process in a spawn context. This is not supported. Please use the same context to create multiprocessing objects and Process.

ANd why it does not crash on MacOS, since Spawn is used by default. Probably, the faulty behavior was found out between Python3.9 and 3.12, fixed to raise a proper, early error, before one tries to actually use the queue, and not back-ported to 3.9.

Indeed, changing both the Queue and Process to start from the "spawn" context works:

import multiprocessing as mp
import time
import traceback
from typing import Any, Optional

import psutil

base_file = "logs.txt"


def main() -> None:
    mp_ctx = mp.get_context("spawn")
    queue: Any = mp_ctx.Queue()
    print("Queue created")
    print("Starting producer process")

    p = mp_ctx.Process(target=producer, args=(queue,), daemon=True)
    p.start()
    print(f"Main: producer started: {p.pid}")

    alive = True
    while alive:
        alive = p.is_alive()
        print(f"Ha ha ha staying alive, producer: {p.is_alive()}")
        time.sleep(1)

    print("Every process is dead :( ")

...

(As a side note, spawn is likely due to become the default multiprocessing behavior even on Linux, as people recently found out some hard-to-debug race conditions that arise in multithreading programs that fork)