Search code examples
apache-kafka

Serialization in Kafka

If I serialize an object manually as a byte array before producing it in kafka will it be serialized internally by kafka again or the kafka's serialization will be skipped on receiving a byte array.

I have done message pack serialization, which is most optimized, I need to know whether my manual serialization is redundant and un-necessary, or this approach is actually useful.

Solution

  • I don't think it's redundant.

    In kafka, Serializer/Deserializer must be specified. ByteArraySerializer (byte array serializer), StringSerializer (string serializer), and IntegerSerializer (integer serializer) are provided by default.

    If you serialize an object manually as a byte array, ByteArraySerializer is a good choice. Your message won't be serialized internally by kafka again.

    But you can custom Serializer by implementing

    org.apache.kafka.common.serialization.Serializer

    which is the best practice.