I am working on a project which uses python and serializes data using avro. Since schema is complex I used avro-gen to generate classes using avro-schema. Now when I am creating and populating these object. My app becomes very slow, it generating like 1000 message per 20 seconds. I have around 200k message which I want to serialize and publish to kafka.
How can I improve the performance of my application? I cannot avoid using avro-gen and if this can be potential bottle neck?
Have you tried using fastavro? It is usually faster than the standard python avro library.