Search code examples
jsonf#-datadeedle

Expanding JSON to columns of a deedle dataframe


I'm trying to convert JSON I get from a website to a deedle dataframe, expanding the JSON entries to separate columns of the dataframe. I found this discussion, but I can't make the proposed solution work for me. As I'm new to both JSON and deedle, I might be making a silly mistake. I'm trying the following (mostly copied from the cited discussion):

let rec expander key value =
    seq {
        match value with
        | JsonValue.String  (s) -> yield key,typeof<string>,box s
        | JsonValue.Boolean (b) -> yield key,typeof<bool>,box b
        | JsonValue.Float   (f) -> yield key,typeof<float>,box f
        | JsonValue.Null    (_) -> yield key,typeof<obj>,box ()
        | JsonValue.Number  (n) -> yield key,typeof<decimal>,box n
        | JsonValue.Record  (r) -> yield! r |> Seq.collect ((<||)expander)
        | JsonValue.Array   (a) ->
            yield! a
            |> Seq.collect (expander "arrayItem")
    }

Frame.CustomExpanders.Add(typeof<JsonDocument>,
                          fun o -> (o :?> JsonDocument).JsonValue |> expander "root")

Frame.CustomExpanders.Add(typeof<JsonValue>,
                          fun o -> o :?> JsonValue |> expander "root")

let info =
    JsonValue.Parse(""" { "name": "Tomas", "born": 1985 } """)

let df =
    [ series ["It" => info] ]
    |> Frame.ofRowsOrdinal

let dfexpanded = Frame.expandAllCols 2 df

This gives me something I don't know how to interpret, but not the desired result:

It.properties                                         It.Tag It.IsString It.IsNumber It.IsFloat It.IsRecord It.IsArray It.IsBoolean It.IsNull It._Print                             
0 -> System.Tuple`2[System.String,FSharp.Data.JsonValue][] 3      False       False       False      True        False      False        False     { "name": "Tomas", "born": 1985 } 

I'm thankful for any input!


Solution

  • The problem seems to be that the type of It in the data frame is not JsonValue, but one of the sub-classes generated by the compiler to represent individual cases of the discriminated union - in this particular case, a nested type named JsonValue+Record.

    Deedle looks for the exact type match (and does not try to find expander for a base class), so a workaround is to register expander for each nested class:

    for t in typeof<JsonValue>.GetNestedTypes() do
      Frame.CustomExpanders.Add(t, fun o -> o :?> JsonValue |> expander "root")
    

    After running this, your code gives the expected result:

    val dfexpanded : Frame<int,string> =
    
         It.name It.born 
    0 -> Tomas   1985