Search code examples
pythonetlpipelineordereddictdagster

Dagster pass ordered dict in the config schema


How can I pass an Ordered Dict in Dagster solid's config schema?

The simple thing:

from dagster import solid, execute_solid, Field


@solid(config_schema={'my_dict': Field(dict, is_required=True)})
def test_ordered_dict(context):
    print('\n\n>>>>>>>>>>>>>>>>>>>>>>>>>>>')
    print(context.solid_config['my_dict'])
    print('\n\n<<<<<<<<<<<<<<<<<<<<<<<<<<<')


for i in range(20):
    execute_solid(test_ordered_dict,
                  run_config={
                      'solids': {
                          'test_ordered_dict': {
                              'config': {
                                  'my_dict': {
                                      'a': 1,
                                      'b': 2
                                  }
                              }
                          }
                      }
                  })

doesn't work. Executing this on Windows 10 consistently gives

2021-03-31 18:16:57 - dagster - DEBUG - ephemeral_test_ordered_dict_solid_pipeline - 8e87a380-7ee5-40b6-954e-3ae33b5784e7 - 26348 - test_ordered_dict - STEP_START - Started execution of step "test_ordered_dict".


>>>>>>>>>>>>>>>>>>>>>>>>>>>
{'b': 2, 'a': 1}


<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021-03-31 18:16:57 - dagster - DEBUG - ephemeral_test_ordered_dict_solid_pipeline - 8e87a380-7ee5-40b6-954e-3ae33b5784e7 - 26348 - test_ordered_dict - STEP_OUTPUT - Yielded output "result" of type "Any". (Type check passed).

On Linux, it sometimes switches them and this doesn't work for me.

How can I pass collections.OrderedDict in the config schema, so that the entries in the dictionary are ordered exactly as I passed them in the schema?


Solution

  • This was a bug that has been fixed https://github.com/dagster-io/dagster/discussions/3977