Search code examples
apache-flinkamazon-kinesis

Flink: Reading from Kinesis causes ReadProvisionedThroughputExceeded

I've got a Flink app with a Kinesis source. I'm seeing a lot of ReadProvisionedThroughputExceeded errors from AWS when running the Flink app. I've tried updating the consumer config with different settings to reduce the number of get record calls and increase time between calls but that doesn't seem to help:

    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_MAX, "500")
    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_INTERVAL_MILLIS, "30000")

    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_BACKOFF_BASE, "3000")
    consumerConfig.put(ConsumerConfigConstants.SHARD_GETRECORDS_BACKOFF_MAX, "10000")

Are there other setting that I should be tuning? Thanks!

Solution

  • Turns out it's a bug: https://issues.apache.org/jira/browse/FLINK-21661 in the version of I'm using - Flink 1.12