dynamoDBStreams getRecords() returning empty records when ShardIteratorType is set to LATEST

Question

I was trying to retrieve last minute's changes in DynamoDB via dynamodb streams with AWS-SDK for nodeJS. I thought of its implementation by using ShardIteratorType as "LATEST". Every time I call getRecords() with the Iterator that I got getShardIterator() it is returning empty records whereas changing ShardIteratorType to "TRIM_HORIZON" returns the objects in that Shard.

I am unable to understand this behaviour. If anyone can help me figure it out, it would be appreciated.

Here is the pseudo code for the implementation.

let getShardIterator = async (shardID) => {
    let params = {
        ShardId: shardID,
        ShardIteratorType: "LATEST", //using TRIM_HORIZON is working fine
        StreamArn: streamInfo.StreamArn
    };
    return new Promise((resolve, reject) => {
        dynamoStreams.getShardIterator(params, function (err, data) {
            if (err) reject(err) // an error occurred
            else resolve(data);
        });
    })
}

let getDataFromIterator = async (params) => {
    return new Promise((resolve, reject) => {
        dynamoStreams.getRecords(params, function (err, data) {
            if (err) reject(err, err.stack); 
            else resolve(data);           
        });
    })
}

let shardIterObj = await getShardIterator(ShardId); 
let data = await getDataFromIterator(shardIterObj); 

Thanks.

Answer 1

As per the docs, LATEST will return just after most recent record, this option normally is used if you are have worker/thread running in background, so that any subsequent events can be picked up.

TRIM_HORIZON - Start reading at the last untrimmed record in the shard in the system, which is the oldest data record in the shard.

LATEST - Start reading just after the most recent record in the shard, so that you always read the most recent data in the shard.