Avro Schema, refer enum values from json file

Question

I'm defining an avro schema with an Enum field for a kafka topic. The avro schema will be uploaded to the kafka schema registry.

I do have a json file in a github repository defined as followed:

validValues.json

{
  "valueId": "abcd123",
  "values":[
     {
       "value": "ENUM1",
       "validDateStart": "2024-01-01",
       "validDateEnd": "2024-12-31"
     },
     {
       "value": "ENUM2",
       "validDateStart": "2024-01-01",
       "validDateEnd": "2024-12-31"
     },
     ....etc
   ]
}

From my Avro file, is the any possibility to refer the enum value from this validValues.json file? I would like to have as values of the Enum fields in the avro file, the values ENUM1, ENUM2 ...etc defined in this json file.

Note that there are more than 100 values in the json file and there can change depending on the application configuration. If there a changed, i want tje change to be reflected in the avro file.

Thanks a lot. Thanks a lot.

Answer 1

You can dynamically construct an Avro schema from your JSON:

private static LocalDate getDate(ObjectNode jsObj, String fieldName) {
    return LocalDate.parse(jsObj.get(fieldName).textValue(), DateTimeFormatter.ISO_LOCAL_DATE);
}

private static Schema generateSchema(ObjectNode enumDescriptor, LocalDate today) {
    return SchemaBuilder.enumeration(enumDescriptor.get("valueId").textValue())
            .symbols(
                    StreamSupport.stream(enumDescriptor.get("values").spliterator(), false)
                            .map(ObjectNode.class::cast)
                            .filter(evd -> !today.isBefore(getDate(evd, "validDateStart")))
                            .filter(evd -> getDate(evd, "validDateEnd").isAfter(today))
                            .map(evd -> evd.get("value").textValue())
                            .toArray(String[]::new)
            );
}

Sample usage:

private static final String JSON = "{\n" +
        "  \"valueId\": \"abcd123\",\n" +
        "  \"values\":[\n" +
        "     {\n" +
        "       \"value\": \"ENUM1\",\n" +
        "       \"validDateStart\": \"2024-01-01\",\n" +
        "       \"validDateEnd\": \"2024-12-31\"\n" +
        "     },\n" +
        "     {\n" +
        "       \"value\": \"ENUM2\",\n" +
        "       \"validDateStart\": \"2024-01-01\",\n" +
        "       \"validDateEnd\": \"2024-12-31\"\n" +
        "     }\n" +
        "   ]\n" +
        "}";

public static void main(String[] args) throws JsonProcessingException {
    final ObjectNode enumJson = (ObjectNode)(new ObjectMapper().readTree(JSON));
    final Schema schema = generateSchema(enumJson, LocalDate.now());
    System.out.println(schema);
}

which outputs the following:

{"type":"enum","name":"abcd123","symbols":["ENUM1","ENUM2"]}