In Azure Event Hubs, what is the simplest way to copy events from one Event Hub to another

Question

I want to copy events including their header data unchanged to another Event Hub.

What I tried so far:

• an Azure Function with an Event Hubs Trigger and an Event Hubs Output. The function was implemented in C#, because that's the only runtime I found where I get access to the headers. The problem I saw here is that when headers are of type byte[], the function fails on the output side with a message that it cannot serialize them. The messages are written to the source Event Hub with Kafka, which means all headers will be of type byte[].
• A simple Spring Cloud Stream application deployed to our OpenShift cluster. This works, but means an extra deployment to operate when we would have liked to have a serverless solution.

Are there simpler ways to do this?

Answer 1

There is a set of Event replication tasks for Azure Functions which are intended to do the translation work and make forwarding events to a second Event Hub easy.

That said, I do not know if it supports maintaining the partition key when doing so - you'd want to test that out to be sure. If not, you would need to manipulate the underlying AMQP Message to attach it.

To do so, you'd call the GetRawAmqpMessage on your destination EventData instance. On the AmqpAnnotatedMessage that gets returned, you'd inject the partition key into the Message Annotations section manually by adding an item with the key x-opt-partition-key and value of the partition key that you'd like it to reflect.

If the replication tasks don't meet your needs for some reason, the best approach would likely be manually publishing events using the method that is discussed in this answer.