I am trying to configure a Kafka source connector to read messages from IBMMQ and route those messages to a Kafka topic. I am getting 2 types of messages, and value of the both message is a string.
Eg: AB ABC123 PQRZ 20220724 IJK BLA BLA-BLA and
AB PQR123 1234 20220724 IJK BLA BLA-BLA XYZ
I want to filter out one of those messages based on the string content of the value.
If I take the above 2 messages as an example, I want the message which contains ABC123 to be produced to the Kafka topic and PQR123 to be dropped. The examples shown in the confluent are all based on JSON payload. Is there any way I can do this without writing a custom transform class (java)
You will need to write custom code, or pre-process the data using tools like Kafka Streams, ksqlDB, Flink, Spark, etc.
Kafka Connect works best on structured data, not plain strings, and even if you have JSON, there is no "String contains" filter that is built-in. The linked transform from Confluent would have to be installed separately, and you can try using =~ JSONPath operator.