Eagerly loading all paginated data with json type provider

Question

Cosider an API which reply is always of this structure:

{
    "pagination": {
        "limit": int,
        "offset": int,
        "count": int,
        "total": int
    },
    "data": [
        {...some obj...}
    ]
}

So payloads differ only in structure of data objects.

Ideally I'd like to tell F# that all types built from samples have some common part - pagination info, so I can have one generic method which reads all pages.

Is it possible, or do I have to extract pagination object and data array separately with two type providers? I see the benefit of having one provider per response body as it supports reading data from the stream.

Answer 1

I would define two different provided types, one for parsing the pagination data and one for parsing the actual data, i.e. something like this:

type Pagination = JsonProvider<"""{ 
  "pagination": { "limit": 1, "offset": 2, 
    "count": 3, "total": 4 } 
  }""">
type OneDataType = JsonProvider<"""{
    "data": [ {"a": 1} ]
  }""">

If you want to avoid parsing the same JSON file twice (e.g. by calling Pagination.Parse and OneDataType.Parse on the same string), you can actually just parse the data once and then pass the parsed JsonValue to the other type:

let odt = OneDataType.Load("/some/file")
let pg = Pagination.Root(odt.JsonValue)
pg.Pagination.Count

If you wanted to do this with a single provided type, then you could define multiple different fields for the multiple different types of data - but you'd have to name those differently. You'd then need to do some fiddling to read the data correctly. I would not do this, because I find it confusing, but it would look something like this:

type AnyData = JsonProvider<"""{ 
  "pagination": { "limit": 1, "offset": 2,
      "count": 3, "total": 4 },
  "data": [],
  "one_data_type": [ {"a":1} ],
  "another_data_type": [ {"b":"xx" }]
}""">

let a = AnyData.Load("/some/file")
// Access pagination data
a.Pagination
// Access data as if it was OneDataType
let oneData = [| for d in a.Data -> 
  AnyData.OneDataType(d.JsonValue) |]