Search code examples
apache-kafkakafka-consumer-apiapache-kafka-streamskafka-producer-api

Using kafka streams trying to write messages from input topic to output topic


Below is my code logic trying to write data from one topic to another.

//Computational logic
StreamsBuilder builder = new StreamsBuilder();
KStream<String, String> kStream = builder.stream(topicName_In);
kStream.foreach((k, v) -> System.out.println("Key = " + k + " Value = " + v));
//kStream.peek((k, v) -> System.out.println("Key = " + k + " Value = " + v));
kStream.to(topicName_Out);
Topology topology = builder.build();

Input Topic data format: Simple Message-1

Error

Exception in thread "HelloStreams-564343a1-1709-4bae-8fe5-514b37eee595-StreamThread-1" org.apache.kafka.streams.errors.StreamsException: Deserialization exception handler is set to fail upon a deserialization error. If you would rather have the streaming pipeline continue after a deserialization error, please set the default.deserialization.exception.handler appropriately.
    at org.apache.kafka.streams.processor.internals.RecordDeserializer.deserialize(RecordDeserializer.java:80)
    at org.apache.kafka.streams.processor.internals.RecordQueue.updateHead(RecordQueue.java:175)
    at org.apache.kafka.streams.processor.internals.RecordQueue.addRawRecords(RecordQueue.java:112)
    at org.apache.kafka.streams.processor.internals.PartitionGroup.addRawRecords(PartitionGroup.java:162)
    at org.apache.kafka.streams.processor.internals.StreamTask.addRecords(StreamTask.java:765)
    at org.apache.kafka.streams.processor.internals.StreamThread.addRecordsToTasks(StreamThread.java:943)
    at org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:764)
    at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:697)
    at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:670)
Caused by: org.apache.kafka.common.errors.SerializationException: Size of data received by IntegerDeserializer is not 4

Solution

  • Given this line

    KStream<String, String> kStream = builder.stream(topicName_In);
    

    I assume that your input data is Strings.

    The error message says

    Size of data received by IntegerDeserializer is not 4
    

    This indicates, that you did configure IntegerSerde (either for the key and/or the value) but you would need to configure StringSerde to make it work.