Multiprocessing within infinite while loop

Question

I wanted to use multiprocessing package to read data with JSON from multiple sources. I also wanted to do it indefinitely.

Here is my code text :

while True:
    time_start = datetime.datetime.now()
    f = open("data_logging_pv.csv", "ab")
    c = csv.writer(f)
    if __name__ == '__main__':
        p=Pool(6)
        output = p.map(getData, [2, 4, 5, 6, 7, 9])
        j = 0
        for i in [2, 4, 5, 6, 7, 9]:
            c.writerow([time_start, i, output[j][0], output[j][1], output[j][2], output[j][3], output[j][4]])
            j = j + 1;
        print(output)
        print("\nTemps d'execution:" +str(datetime.datetime.now()-time_start))
        f.close()
        p.terminate()

Where the function getData is the JSON request.

The execution does not realize the p.map, but the processes are wel created. I have no clue how to debug this.

I'm a bit rusty with python and have enver used multiprocessing, then there may be some 'basic' errors.

Thanks Martin.

Answer 1

1. Added KeyboardInterrupt handler, so it wont spam exceptions into stdout
2. Little optimized csv writing
3. Pool initialized only once

Not sure, that kind of errors did you have, but this snippet works propely on python2.7:

import csv
import datetime
import signal
from multiprocessing.pool import Pool


def getData(item):
    return [item, item + 1, item ** item, item * 2, 0]


def worker():
    signal.signal(signal.SIGINT, signal.SIG_IGN)


if __name__ == '__main__':

    time_start = datetime.datetime.now()
    file = open("data_logging_pv.csv", "ab")
    c = csv.writer(file)
    pool = Pool(6, initializer=worker)

    try:

        while True:
            output = pool.map(getData, [2, 4, 5, 6, 7, 9])

            for i, res in zip([2, 4, 5, 6, 7, 9], output):
                c.writerow([str(datetime.datetime.now()), i] + res)

    except KeyboardInterrupt:
        exit()
    finally:
        pool.terminate()
        pool.join()
        print("\nTemps d'execution:" + str(datetime.datetime.now() - time_start))
        file.close()