Search code examples
pythonmultiprocessingcprofile

Python multiprocess profiling

I want to profile a simple multi-process Python script. I tried this code:

import multiprocessing
import cProfile
import time
def worker(num):
    time.sleep(3)
    print 'Worker:', num

if __name__ == '__main__':
    for i in range(5):
        p = multiprocessing.Process(target=worker, args=(i,))
        cProfile.run('p.start()', 'prof%d.prof' %i)

I'm starting 5 processes and therefore cProfile generates 5 different files. Each log only shows what happens inside the start method. How can I get logs that profile the worker function (and show that it took approximately 3 seconds in each case)?

Solution

  • You're profiling the process startup, which is why you're only seeing what happens in p.start() as you say—and p.start() returns once the subprocess is kicked off. You need to profile inside the worker method, which will get called in the subprocesses.