Is there a way to create an "Alias" or a "soft-link" of a subject in schema registry? I'm asking this is because for certain kafka topics, they share a common schema e.g. topic names:
{env}.{level}.{schema_name}.{producer_name}.{realtime|batch}
schema subject:
{env}.{level}.{schema_name}.{version}
When I set up confluent s3 sink connector, the connector avro converter follows topicNameStrategy to derive schema subject from topic name so the schema name is derived to be {topicName}-value
Is there a way for me to create alias {topicName}-value which actually points to {env}.{level}.{schema_name}.{version} in schema registry? I'm trying to avoid duplicating schemas in schema registry or any significant changes in current kafka topics.
Thanks.
When I set up confluent s3 sink connector, the connector avro converter follows topicNameStrategy
You can set value.converter.value.subject.name.strategy to change that.
But that means you need to write your own class to get custom one not offered by Confluent. Sounds like your producer's are already using the same class to register those formats in the Registry rather than also the default {topic}-value, anyway.
The only way to "alias" is to explicitly make the HTTP requests to PUT a new version of a new subject after GET request against the "original". You aren't duplicating anything more than metadata since the schema will remain the same. The main downside is that there is no "linking" between the "aliases" since such a concept doesn't really exist, and therefore, you need to remember to change all subjects.
Personally, I used the pattern of hard-coding env=DEV/level=FOO/ in the topics.dir setting, then the topic names were underneath that. If you have dynamic values, then put those fields in the record itself, and use S3 Connect's FieldPartitioner to accomplish the same thing