Python multiple TCP Socket Client via multiprocessing

Question

so, I'm trying to build the following setup:

• have a TCP server that handles incoming connections via new threads,
• have another process, that creates new proccesses (TCP Clients) that connect to that TCP server.

I am using Python 3.11.2 on a Windows 10 laptop.

For the most parts, it works just fine. I am only sometimes having problems with the TCP Clients proccesses. Problem is that some proccesses dont't get created. This cannot be reproduced all the time, sometimes I have the problem when creating 5 client-proccesses, sometimes it runs smoothly when creating 150. The TCP server socket.listen() backlog is sufficiently high.

Server code

import threading
import socket

MAX_DATASIZE = 2048;

HOST = "127.0.0.1"
PORT = 5000

def handleSingleSocket(conn, addr):
    print("Client "+ str(addr[0]) + ": " + str(addr[1]) + " connected")
    data = conn.recv(MAX_DATASIZE)
    print(data)
    conn.sendall(data)
    conn.close()
    print("Client "+ str(addr[0]) + ": " + str(addr[1]) + " disconnected")

listSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listSocket.bind((HOST, PORT))
listSocket.listen(200)
print(f"Server listening on address {HOST} and port {PORT}")

while True:
    conn, addr = listSocket.accept()
    thread = threading.Thread(target=handleSingleSocket,args=(conn,addr))
    thread.start()
    print(f"[ACTIVE CONNECTIONS]: {threading.active_count() - 1}" )

Client Code (sleep is to have all the clients connected at the same time)

import socket
import multiprocessing
import time

MAX_DATASIZE = 2048;
HOST = "127.0.0.1"
PORT = 5000

def handleSingleConnection(sock: socket.socket, i):
    sock.connect((HOST, PORT))
    time.sleep(2)
    sock.sendall(f"Hello from Process {i}!".encode())
    result = sock.recv(MAX_DATASIZE)
    print(result)
    sock.close()

if (__name__ == '__main__'):
    for i in range(150):
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        process = multiprocessing.Process(target=handleSingleConnection, args=(sock,i,))
        process.start()

Some connections don't get created. The error code is something like that:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 120, in spawn_main 
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\spawn.py", line 130, in _main      
    self = reduction.pickle.load(from_parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\reduction.py", line 238, in _rebuild_socket
    return ds.detach()
           ^^^^^^^^^^^
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\resource_sharer.py", line 38, in detach
    with _resource_sharer.get_connection(self._id) as conn:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\resource_sharer.py", line 86, in get_connection
    c = Client(address, authkey=process.current_process().authkey)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\connection.py", line 499, in Client    c = PipeClient(address)
        ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\erath\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\connection.py", line 701, in PipeClient
    _winapi.WaitNamedPipe(address, 1000)
FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden

So, I guess it has to do with the spawn creation of proccesses on Windows, but I can't really wrap my head around it. Help is really appreciated, thanks in advance! :)

Answer 1

https://superfastpython.com/filenotfounderror-multiprocessing-python/

With the help of this article, I was able to fix the problem.

Apparently, the main function exits before the child proccesses do, so the files in the library cannot be found.

The article defines three preconditions for this kind of behaviour:

1. Use the ‘spawn’ start method to create new processes.
2. Share a concurrent primitive with one or more processes.
3. Main process exits before the child process starts up fully.

Interestingly, I was not using a concurrent primitive and still faced this problem.

Appending the proccesses in a list before starting and then joining them after their execution in the main function, did the trick.

if __name__ == '__main__':
    plist = []
    for i in range(200):
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        process = multiprocessing.Process(target=handleSingleConnection, args=(sock,i))
        plist.append(process)
        process.start()
    for p in plist:
        p.join()