Spring Cloud Aws kinesis Binder All JVMs in group consume same message at different intervals

Question

We have three containers connecting to Kinesis stream as consumers using Spring cloud AWS kinesis binder. We have used consumer groups for load balancing across three containers. The requirement is that containers share the load and it shall be possible to distribute the load evenly.

1. All containers use the same application config from config server

Currently we have configuration like below.

spring:
  cloud:
    stream:
      bindings:
        MyStream:
          group: my-group
          destination: stream-1
          content-type: application/json

All containers are consuming same message, but at the different times (having difference of 5 to 10 mins)

As per the documentation,

Static shard distribution within a single consumer group It is possible to evenly distribute shard across all instances within a single consumer group. This done by configuring:

spring.cloud.stream.instanceCount= to number of instances

spring.cloud.stream.instanceIndex= current instance’s index

As we use the same config server, can you please help how to make sure that load balancing is achieved.

Answer 1

First of all if you use DynamoDB in your project, a DynamoDbLockRegistry bean will be created to allow all the instance to get access to a shared data about exclusive access to every shard. Therefore only one instance will be able to consume records from one shard.

It is possible that only one instance will pick up all the shards from a stream, but it doesn't look like the same record may go another instance.

For the spring.cloud.stream.instanceIndex you can feed it as a JVM arg: -Dspring.cloud.stream.instanceIndex= for each instance as unique value, so each instance will get only its own subset of shards from the stream. Otherwise it isn't possible to distinguish it via common config server.