Search code examples
javamavenapache-kafkamaven-pluginconfluent-schema-registry

Using kafka-schema-registry-maven-plugin with parent and child schemas


I have a project where there are three schemas MetaModel.avsc, RevisionModel.avsc and RecentChangeEventModel.avsc. Where RecentChangeEventModel refers to the others as nested objects like the following:

{
  "namespace": "org.taulin.model",
  "type": "record",
  "name": "RecentChangeEvent",
  "fields": [
    {
      "name": "schema",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "meta",
      "type": [
        "null",
        "Meta"
      ],
      "default": null
    },
    {
      "name": "id",
      "type": [
        "null",
        "long"
      ],
      "default": null
    },
    {
      "name": "type",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "namespace",
      "type": [
        "null",
        "int"
      ],
      "default": null
    },
    {
      "name": "title",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "titleUrl",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "comment",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "timestamp",
      "type": [
        "null",
        "long"
      ],
      "default": null
    },
    {
      "name": "user",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "bot",
      "type": [
        "null",
        "boolean"
      ],
      "default": null
    },
    {
      "name": "notifyUrl",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "serverUrl",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "serverName",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "serverScriptPath",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "wiki",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "parsedComment",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },
    {
      "name": "minor",
      "type": [
        "null",
        "boolean"
      ],
      "default": null
    },
    {
      "name": "patrolled",
      "type": [
        "null",
        "boolean"
      ],
      "default": null
    },
    {
      "name": "length",
      "type": [
        "null",
        "Revision"
      ],
      "default": null
    },
    {
      "name": "revision",
      "type": [
        "null",
        "Revision"
      ],
      "default": null
    }
  ]
}

And I have the following configuration with the kafka-schema-registry-maven-plugin:

            <plugin>
                <groupId>io.confluent</groupId>
                <artifactId>kafka-schema-registry-maven-plugin</artifactId>
                <version>7.7.1</version>
                <configuration>
                    <schemaRegistryUrls>
                        <param>http://localhost:8081/</param>
                    </schemaRegistryUrls>
                    <subjects>
                        <Meta>${project.basedir}/src/main/avro/MetaModel.avsc</Meta>
                        <Revision>${project.basedir}/src/main/avro/RevisionModel.avsc</Revision>
                        <RecentChangeEvent>${project.basedir}/src/main/avro/RecentChangeEventModel.avsc</RecentChangeEvent>
                    </subjects>
                    <schemaTypes>
                        <Meta>AVRO</Meta>
                        <Revision>AVRO</Revision>
                        <RecentChangeEvent>AVRO</RecentChangeEvent>
                    </schemaTypes>
                    <references>
                        <order>
                            <reference>
                                <name>Meta</name>
                                <subject>Meta</subject>
                            </reference>
                            <reference>
                                <name>Revision</name>
                                <subject>Revision</subject>
                            </reference>
                            <reference>
                                <name>RecentChangeEvent</name>
                                <subject>RecentChangeEvent</subject>
                            </reference>
                        </order>
                    </references>
                </configuration>
                <goals>
                    <goal>register</goal>
                </goals>
            </plugin>

When I try executing mvn schema-registry:register, the following error occurs:

[ERROR] Could not parse Avro schema
org.apache.avro.SchemaParseException: Undefined name: "org.taulin.model.Meta"
    at org.apache.avro.Schema.parse (Schema.java:1697)
    at org.apache.avro.Schema.parse (Schema.java:1823)
    at org.apache.avro.Schema.parse (Schema.java:1736)
    at org.apache.avro.Schema$Parser.parse (Schema.java:1471)

I have tried changing plugin configurations and the way that each schema is referred but nothing seems to work. It seems that the plugin was not made to be used when schemas refers to each other in this manner.


Solution

  • The problem was that references tag was wrongfully structured. In this case RecentChangeEvent should be the parent tag and the referenced objects/schemas should be nested like the following:

    <references>
        <RecentChangeEvent>
            <reference>
                <name>Meta</name>
                <subject>Meta</subject>
            </reference>
            <reference>
                <name>Revision</name>
                <subject>Revision</subject>
            </reference>
        </RecentChangeEvent>
    </references>