How to merge a dynamically named record with a static one in Dhall?

I'm creating an AWS Step Function definition in Dhall. However, I don't know how to create a common structure they use for Choice states such as the example below:

{
    "Not": {
        "Variable": "$.type",
        "StringEquals": "Private"
    },
    "Next": "Public"
}

The Not is pretty straightforward using mapKey and mapValue. If I define a basic Comparison:

{ Type =
    { Variable : Text
    , StringEquals : Optional Text
    }
, default = 
    { Variable = "foo" 
    , StringEquals = None Text
    }
}

And the types:

let ComparisonType = < And | Or | Not >

And adding a helper function to render the type as Text for the mapKey:

let renderComparisonType = \(comparisonType : ComparisonType )
    -> merge
    { And = "And"
    , Or = "Or"
    , Not = "Not"
    }
    comparisonType

Then I can use them in a function to generate the record halfway:

let renderRuleComparisons = 
  \( comparisonType : ComparisonType ) ->
  \( comparisons : List ComparisonOperator.Type ) ->
    let keyName = renderComparisonType comparisonType
    let compare = [ { mapKey = keyName, mapValue = comparisons } ]
    in compare

If I run that using:

let rando = ComparisonOperator::{ Variable = "$.name", StringEquals = Some "Cow" }
let comparisons = renderRuleComparisons ComparisonType.Not [ rando ]
in comparisons

Using dhall-to-json, she'll output the first part:

{
    "Not": {
        "Variable": "$.name",
        "StringEquals": "Cow"
    }
}

... but I've been struggling to merge that with "Next": "Sup". I've used all the record merges like /\, //, etc. and it keeps giving me various type errors I don't truly understand yet.

Solution

First, I'll include an approach that does not type-check as a starting point to motivate the solution:

let rando = ComparisonOperator::{ Variable = "$.name", StringEquals = Some "Cow" }

let comparisons = renderRuleComparisons ComparisonType.Not [ rando ]

in  comparisons # toMap { Next = "Public" }

toMap is a keyword that converts records to key-value lists, and # is the list concatenation operator. The Dhall CheatSheet has a few examples of how to use both of them.

The above solution doesn't work because # cannot merge lists with different element types. The left-hand side of the # operator has this type:

comparisons : List { mapKey : Text, mapValue : Comparison.Type }

... whereas the right-hand side of the # operator has this type:

toMap { Next = "Public" } : List { mapKey : Text, mapValue : Text }

... so the two Lists cannot be merged as-is due to the different types for the mapValue field.

There are two ways to resolve this:

Approach 1: Use a union whenever there is a type conflict
Approach 2: Use a weakly-typed JSON representation that can hold arbitrary values

Approach 1 is the simpler solution for this particular example and Approach 2 is the more general solution that can handle really weird JSON schemas.

For Approach 1, dhall-to-json will automatically strip non-empty union constructors (leaving behind the value they were wrapping) when translating to JSON. This means that you can transform both arguments of the # operator to agree on this common type:

List { mapKey : Text, mapValue : < State : Text | Comparison : Comparison.Type > }

... and then you should be able to concatenate the two lists of key-value pairs and dhall-to-json will render them correctly.

There is a second solution for dealing with weakly-typed JSON schemas that you can learn more about here:

Dhall Manual - How to convert an existing YAML configuration file to Dhall

The basic idea is that all of the JSON/YAML integrations recognize and support a weakly-typed JSON representation that can hold arbitrary JSON data, including dictionaries with keys of different shapes (like in your example). You don't even need to convert the entire the expression to this weakly-typed representation; you only need to use this representation for the subset of your configuration where you run into schema issues.

What this means for your example, is that you would change both arguments to the # operator to have this type:

let Prelude = https://prelude.dhall-lang.org/v12.0.0/package.dhall

in  List { mapKey : Text, mapValue : Prelude.JSON.Type }

The documentation for Prelude.JSON.Type also has more details on how to use this type.