Search code examples
daskdask-distributeddask-delayed

Passing Futures as arguments in Dask

What is the best way to pass a Future to a Dask Delayed function such that the Future stays in tact? In other words, how can we ensure the function will get the actual Future and not the result it represents?

Solution

  • Generally the semantics are that dask.delayed functions get concrete results rather than dask-y ones. This is not easily supported today without some tricks.

    That being said, I recommend the following trick:

    Place your future in a Variable

    import dask
from dask.distributed import Client, Variable
client = Client()

v = Variable()

@dask.delayed
def f(v):
    return v.get().result()

future = client.scatter(123)
f(future).compute()
# 123