Kafka - how to transform Rest service message to ingest into a kafka topic?

Question

I'm new at kafka community and i'm facing a challenging question. I have two applications that talk each other via Rest webservice, with the body being a json message.

How can I use kafka to be the middleware between these two application, with the minor or zero impact on the applications?

This is my As-Is scenario:

My To-Be scenario:

I've thought in use STM or some interceptor to transform the header and body.

Is it possible to do that way? I know that I have to transform the body from:

      {
        "first name" : "Donald" ,
        "last name" : "Trump"
      },
      {
        "first name" : "Vladimir" ,
        "last name" : "Putin"
      }

to:

{
  "records":
  [
    {
      "key": "K001",
      "value":
      {
        "first name" : "Donald" ,
        "last name" : "Trump"
      }
    },
    {
      "key": "K002",
      "value":
      {
        "first name" : "Vladimir" ,
        "last name" : "Putin"
        }
    }
  ]
}

Essentially my transformation would change the beginning and the end of the message, and not a field of each record.

Anyone have done something like that, or has a indication of which way should I go?

Answer 1

Outside of the Kafka ecosystem, you can use NiFi or Streamsets for these "message transforms" you are looking for, and both can do HTTP calls as well as starting HTTP servers themselves for direct API consumption.