Search code examples
avroavsc

Avro schema getting undefined type name when using Record type


so im trying to parse an object with this avro schema.

object is like:

myInfo: {size: 'XL'}

But Its behaving like the record type doesn't actually exist and im getting a undefined type name: data.platform_data.test_service.result.record at Function.Type.forSchema for it.

schema looks like:

  "avro": {
    "metadata": {
      "loadType": "full",
      "version": "0.1"
    },
    "schema": {
      "name": "data.platform_data.test_service.result",
      "type": "record",
      "fields": [
        {
          "name": "myInfo",
          "type": "record",
          "fields": [{
            "name": "size",
            "type": {"name":"size", "type": "string"}
          }]
      }
      ]
    }
  }

I should mention im also using avsc for this. Anybody have any ideas? I've tried pretty much all combinations but afaik the only way of parsing out an objct like this is with record


Solution

  • Playing around with the schema, I found that "type": "record" is a problem. I moved it to nested definition. And it worked. Seems like description here is little bit confusing.

    Change Before:

    {
      "name": "myInfo",
      "type": "record",
      "fields": [{
        "name": "size",
        "type": {"name":"size", "type": "string"}
      }]
    }
    

    After:

    {
      "name": "myInfo",
      "type": {
        "type": "record",
        "name": "myInfo",
        "fields": [
          {
            "name": "size",
            "type": {"name":"size", "type": "string"}
          }
        ]
      }
    }
    

    Updated schema which is working:

    {
      "name": "data.platform_data.test_service.result",
      "type": "record",
      "fields": [
        {
          "name": "myInfo",
          "type": {
            "type": "record",
            "name": "myInfo",
            "fields": [
              {
                "name": "size",
                "type": {"name":"size", "type": "string"}
              }
            ]
          }
        }
      ]
    }
    

    To make a record attribute nullable, process is same as any other attribute. You need to union with "null" (as show in below schema):

    {
      "name": "data.platform_data.test_service.result",
      "type": "record",
      "fields": [
        {
          "name": "myInfo",
          "type": [
            "null",
            {
              "type": "record",
              "name": "myInfo",
              "fields": [
                {
                  "name": "size",
                  "type": {
                    "name": "size",
                    "type": "string"
                  }
                }
              ]
            }
          ]
        }
      ]
    }