GSI parallel query for vertical scaling of ECS

Question

I have an Application deployed in ECS, which query Dynamodb on the status column using GSI and perform some computations and update the status.

My problem is querying the GSI in pagination takes a long time, I want to scale application horizontally i.e increase the desired count from 1 to 5. Which will launch 5 ecs task of the same application. Doing this makes it difficult to query.

One approach I can think about is adding a sort key to gsi and adding values like 1,2,3,4,5 and then the task definition 1 will query on records with sort key 1 and so on. But the problem here is I cannot think of a query that will achieve the same.

Need help to get thing working. I am doing this in java

Answer 1

We have achieved this by implementing the token management system.

We added an entry in the db with the below attributes

Id: TOKEN
CurrentAvailableToken: 5
MaxToken : 5

We will make the Desired count as 5.

When task-1 runs it will get the CurrentAvailableToken as 5 and decrement and update the token as 4. task-1 will read all data with status=ACTIVE and taskNumber=5.

When task-1 runs it will get the CurrentAvailableToken as 4 and decrement and update the token as 3. task-1 will read all data with status=ACTIVE and taskNumber=4

and it continues.