Search code examples
dagster

Does dagster support dependencies between solids with no outputs?


I am building a prototype pipeline that does two things:

  1. (solid) Clears files out of an existing directory
  2. (solid) Runs a batch process to dump data into that directory.

Step #1 is all side-effect, and has no output to pass to #2. Is it possible to express a dependency between these two solids in a pipeline?


Solution

  • I think the following code snippet from https://docs.dagster.io/examples/nothing should work for your use case:

    from dagster import Nothing
    
    @solid
    def create_table_1(_) -> Nothing:
        get_database_connection().execute("create table_1 as select * from some_source_table")
    
    
    @solid(input_defs=[InputDefinition("start", Nothing)])
    def create_table_2(_):
        get_database_connection().execute("create table_2 as select * from table_1")
    
    
    @pipeline
    def my_pipeline():
        create_table_2(create_table_1())