current setting of KAFKA Consumer:
max_poll_records: 2000
max_poll_interval_ms: 40000
fetch_min_bytes: 104857600
fetch_max_wait_ms: 10000
fetch_max_bytes: 104857600
request_timeout_ms: 300000
max_partition_fetch_bytes: 104857600
heartbeat_interval_ms: 60000
session_timeout_ms: 180000
current setting at KAFKA Broker:
fetch.max.bytes: 104857600
with these settings I am able to poll 2000 messages at once of size 84MB approx (size of one poll). but the second message i am receiving is lesser than before, while lag of sufficinet amount exists on that consumer.
I am pushing to only one partition so that poll also happens from one partition.
As seen in the screenshot of code, messages polled are in a pattern: 1st poll: 2000 messages 2nd poll: 484 messages 3rd poll: 2000 messages 4th poll: 484 messages 5th poll: 2000 messages 6th poll: 484 messages 7th poll: 548 messages and so on
Can anyone help me figure this out, why is this happening?
I tried searching for the solution on google but didn't find much
It is limited by max.partition.fetch.bytes,
MAX.PARTITION.FETCH.BYTES
This property controls the maximum number of bytes the server will return per partition. The default is 1 MB, which means that when KafkaConsumer.poll() returns ConsumerRecords, the record object will use at most max.partition.fetch.bytes per partition assigned to the consumer. So if a topic has 20 partitions, and you have 5 consumers, each consumer will need to have 4 MB of memory available for ConsumerRecords. In practice, you will want to allocate more memory as each consumer will need to handle more partitions if other consumers in the group fail. max. partition.fetch.bytes must be larger than the largest message a broker will accept (determined by the max.message.size property in the broker configuration), or the broker may have messages that the consumer will be unable to consume, in which case the consumer will hang trying to read them. Another important consideration when setting max.partition.fetch.bytes is the amount of time it takes the consumer to process data. As you recall, the consumer must call poll() frequently enough to avoid session timeout and subsequent rebalance. If the amount of data a single poll() returns is very large, it may take the consumer longer to process, which means it will not get to the next iteration of the poll loop in time to avoid a session timeout. If this occurs, the two options are either to lower max. partition.fetch.bytes or to increase the session timeout.