Search code examples
pythonavrofastavro

How can I create an Avro schema from a python class?


How can I transform my simple python class like the following into a avro schema?

class Testo(SQLModel):
    name: str
    mea: int

This is the Testo.schema() output

{
    "title": "Testo",
    "type": "object",
    "properties": {
        "name": {
            "title": "Name",
            "type": "string"
        },
        "mea": {
            "title": "Mea",
            "type": "integer"
        }
    },
    "required": [
        "name",
        "mea"
    ]
}

from here I would like to create an Avro record. This can be converted online on konbert.com (select JSON to AVRO Schema) and it results in the Avro schema below. (all valid despite the name field which should be "Testo" instead of "Record".)

{
  "type": "record",
  "name": "Record",
  "fields": [
    {
      "name": "title",
      "type": "string"
    },
    {
      "name": "type",
      "type": "string"
    },
    {
      "name": "properties.name.title",
      "type": "string"
    },
    {
      "name": "properties.name.type",
      "type": "string"
    },
    {
      "name": "properties.mea.title",
      "type": "string"
    },
    {
      "name": "properties.mea.type",
      "type": "string"
    },
    {
      "name": "required",
      "type": {
        "type": "array",
        "items": "string"
      }
    }
  ]
}

Anyhow, if they can do it, there certainly must be a way to convert it with current python libraries. Which library can do a valid conversion (and also complex python models/classes?

If there is an opinion of that this is a wrong approach, that is also welcome - if - pointing out a better way how this translation process can be done.


Solution

  • It looks like there are a few libraries that aim to provide this kind of functionality:

    1. py-avro-schema has support for generic Python classes
    2. dataclasses-avroschema has support for dataclasses, pydantic models, and faust records
    3. pydantic-avro requires your Python class to inherit from pydantic.BaseModel