Multiprocessing only utilizing a single core

Question

I'm trying to create a FBM texture using the module perlin-noise, but it takes a very long time to execute. I've implemented multiprocessing, only to find that the program was still running off a single core. I've tried looking for other people with the same problem, but most threads were 7+ years old and / or involved problems and solutions related to different OSs.

My OS is Windows 8.1, I have a quad-core CPU, and I'm running Python 3.9.2

Here is the program:

from perlin_noise import PerlinNoise
from multiprocessing import Pool

def world(n):
    noise1 = PerlinNoise(octaves=3, seed=1)
    noise2 = PerlinNoise(octaves=6, seed=1)
    noise3 = PerlinNoise(octaves=12, seed=1)
    noise4 = PerlinNoise(octaves=24, seed=1)
    noise5 = PerlinNoise(octaves=48, seed=1)
        
    world = []
    for i in range(n):
        row = []
        for j in range(n):
            noise_val =  noise1([i/n, j/n])
            noise_val += 0.5 * noise2([i/n, j/n])
            noise_val += 0.25 * noise3([i/n, j/n])
            noise_val += 0.125 * noise4([i/n, j/n])
            noise_val += 0.0625 * noise5([i/n, j/n])
            row.append(noise_val)
        world.append(row)
    
def main():
    start = time.time()
    nums = [128]
    p = Pool()
    p.map(world, nums)
    p.close()
    p.join()
    end = time.time()
    print(end - start)
    
if __name__ == '__main__':
    import time
    from distributed import Client
    client = Client()
    main()

So, what's going on? Have I made a mistake in thinking that multiprocessing could work on these for-loops?

Thanks!

Answer 1

Reason why it only use one Process is simple. You only passed 1-length list in Pool.map.

What Pool(n).map(function, iterable) does is, applying provided funtion to each element of provided iterable(in this case, list) with n number of worker processes.

Since you only have 128 in nums it's only creating one task thus no other processes are ever used.

Proper usage would look like this:

from multiprocessing import pool


def work(val):
    return hash(val ** 10000000)


if __name__ == '__main__':
    p = pool.Pool(8)

    with p:
        output = p.map(work, [i for i in range(1, 31)])

    print(output)

This example will have 8 processes, so utilizing 8 logical cores. Since we gave it numbers from 1 to 30, p.map will apply function work to each of those numbers using 8 processes, so it can run up to 8 functions simultaneously.

When we run it, we can see it's effect.

Of course it's using more processes to communicates between process, etc - I admit that I don't know underlying details on it.

---

Side note, to make your work more efficient, you should refrain yourself from appending as much as possible.

Check this simple example appending a lot of time and measuring time it took.

>>> import timeit
>>> def appending():
...     output = []
...     for i in range(1000000):
...         output.append(i)
...     return output
... 
>>> def list_comp():
...     return [i for i in range(1000000)]

>>> print(f"{timeit.timeit(appending, number=100):.2}")
8.1

>>> print(f"{timeit.timeit(list_comp, number=100):.2}")
5.2

As you can see appending is pretty much more slower than List comprehension - but this doesn't mean not to use list.append - just don't use it excessively.