Search code examples
dagster

Executing a solid when at least one of the required inputs is given

As an input, I would like to retrieve data based on user input, or "randomly" from a DB if no user input is given. All other downstream tasks of the pipeline would be the same.

Therefore, I would like to create a pipeline starting with solids A and B, and a downstream solid C executed based on input from solid A OR solid B.
However, when using conditional outputs on solids A and B, solid C is not executed, as one input is not generated by upstream solids.

Is there a simple way of doing this that I am missing out?

Thanks for your help.

Solution

  • "fan-in" dependencies will not skip unless all the fanned in outputs were skipped, so that is one way to accomplish this.

    @pipeline
def example():
  maybe_a = A()
  maybe_b = B()
  C(items=[maybe_a, maybe_b])

    https://docs.dagster.io/examples/fan_in_pipeline#main