How to input data through Kafka REST in an environment where message payload cannot be edited

Question

I want to send push and pull events of images to Kafka using the Azure ACR Webhook function. 'Event Hub' is not available due to cost.

The webhook is set to be thrown to the Kafka Rest Proxy URI with the header 'Content-Type: application/vnd.kafka.json.v2+json'.

When an event occurs Headers: 'User-Agent: AzureContainerRegistry/1.0 traceparent: 00-1318068fc8e0ca8904b19865b65d551b-d98ef2405fa798bf-01 Content-Type: application/vnd.kafka.json.v2+json; charset=utf-8 Content-Length: 104' Payload: ' { "id": "a00c973c-a717-425d-9db6-874a6288728c", "timestamp": "2024-03-13T06:56:49.2277133Z", "action": "ping" } '

The message is sent, but the response is '{"error_code":422,"message":"Unrecognized field: id"}'.

This seems to be because the payload format of the webhook is sent without the 'records, value' keys as follows. { "records":[{"value": { ACTUAL DATA }}]}

In this environment where the payload format of the message cannot be edited, Is there a solution that matches the message format and sends it to Kafka, or is there a more efficient REST solution that can receive such a simple message and send it to Kafka?

PLUS: Please let me know if there is a configuration that allows you to send messages to Kafka using Azure Webhook without Event Hub. thank you

Answer 1

There is an open-source Kafka proxy called Zilla. It has a yaml config file that you can use to define the REST endpoint you want because it uses only the HTTP payload as the Kafka message payload. It sounds like you want the onway produce feature which. Here is an example.

Obligatory: I am a committer to the Zilla project and work with the team to maintain it.