How to wait for other python processes to put string into queue before putting next item into queue

Question

I have a small example for my problem:

from multiprocessing import Process, Queue

def test_1(q,j):
    for i in range(10):
        q.put('test_{}: '.format(j) + str(i),block=False)



q = Queue()
p1 = Process(target=test_1, args=(q,1))
p2 = Process(target=test_1, args=(q,2))
p1.start()
p2.start()

with open('test1.txt', 'w') as file:
    while p1.is_alive() or p2.is_alive() or not q.empty():
        try:
            value = q.get(timeout = 1)
            file.write(value + '\n')
        except Exception as qe:
            print("Empty Queue or dead process")

p1.join()
p2.join()

Then my output is like:

test_1: 0
test_1: 1
test_1: 2
test_1: 3
test_1: 4
test_1: 5
test_1: 6
test_1: 7
test_1: 8
test_1: 9
test_2: 0
test_2: 1
test_2: 2
test_2: 3
test_2: 4
test_2: 5
test_2: 6
test_2: 7
test_2: 8
test_2: 9

How can I get an output like:

test_1: 0
test_2: 0
test_1: 1
test_2: 1
test_1: 2
test_2: 2
.
.
.

and so on.

Would be cool if somebody could help me with that.

For your interest, I would like to use that later to match input- and output-vectors as training data for a machine learning approach.

Thanks in advance,

Yaz

Answer 1

In general when I want to use multithreading with python I use ``ThreadPool``` I'm not sure that's really what you want but you can easly get your result using this function as written here:

from multiprocessing.pool import ThreadPool
from psutil import cpu_count

def test_1(thread):
    for i in range(10):
        string = ('test_{}: '.format(thread) + str(i))
        print(string)

pool = ThreadPool(processes=cpu_count(logical=True))

lines_async = pool.apply_async(test_1, args=([1]))
lines_async2 = pool.apply_async(test_1, args=([2]))
lines_async.get()
lines_async2.get()

the result is here :

test_1: 0
test_2: 0
test_1: 1
test_2: 1
test_1: 2
test_2: 2
test_1: 3
test_2: 3
test_1: 4
test_2: 4
test_1: 5
test_2: 5
test_1: 6
test_2: 6
test_1: 7
test_2: 7
test_2: 8
test_1: 8
test_2: 9
test_1: 9

but it's a different approach than the one you carried out

EDIT: if you want to get the value of the function you can use it :

from multiprocessing.pool import ThreadPool
from psutil import cpu_count

pool = ThreadPool(processes=cpu_count(logical=True))

def test_1(thread, i):
    string = ('test_{}: '.format(thread) + str(i))
    return string

for i in range(10):
    lines_async = pool.apply_async(test_1, args=([1,i]))
    lines_async2 = pool.apply_async(test_1, args=([2,i]))
    string1 = lines_async.get()
    string2 = lines_async2.get()
    print(string1)
    print(string2)

which is more similar than what you want and gives the same result.