How to make dworkers for multiprocess?

Question

I am working on Distributed cluster computing. To implement such system I am trying to use python libs that is dask.distriuted. But there has a problem that is the dworkers are not for multiprocess, means 2 or 3 dworkers, works together but don't support multiple executions that support in multiprocessing lib.

for an example:

def testFun():
 while True:
  time.sleep(3)
  print('looping')

If I executes this function in the client.submit(testFun).It will execute this function for infinite times then it will never come to the next steps. Like for this program:

client.submit(testFun)
client.submit(testFun)

Here until execute the first line it will never come to the next line. I want to make that dworker for multiprocessing. How will I do this ?

Answer 1

That's because the function has the same signature, only runs one time.

You can tell by the key that is generated. See:

In [5]: client.submit(testFun)
<Future: status: pending, key: testFun-a4102f4653c498f9fafc90003d87bd08>

In [6]: client.submit(testFun)
<Future: status: pending, key: testFun-a4102f4653c498f9fafc90003d87bd08>

Try this

def testFun(x):
    while True:
        time.sleep(3)
        print('looping', x)

In [13]: client.submit(testFun, 1)
<Future: status: pending, key: testFun-afa640a088a357e5f8dd46c1937af3a7>

In [14]: client.submit(testFun, 2)
<Future: status: pending, key: testFun-98309530cb5b26d69131e54a521b8b40>