How to read an individual column from Dynamo-Db without using Scan in Node-js?

Question

I have 4.5 millions of records in my Dynamo Db.

I want to read the the id of each record as a batchwise.

i am expecting something like offset and limit like how we can read in Mongo Db.

Is there any way suggestions without scan method in Node-JS.

I have done enough research i can only find scan method which buffers the complete records from Dynamo Db and the it starts scanning the records, which is not effective in performance basis.

Please do give me suggestion.

Answer 1

First things to know about DynamoDB is that it is a Key-Value Store with support for secondary indexes.

DynamoDB is a bad choice if the application often has to iterate over the entire data set without using indexes(primary or secondary), because the only way to do that is to use the Scan API.

DynamoDB Table Scan's are (a few things I can think off)

1. Expensive(I mean $$$)
2. Slow for big data sets
3. Might use up the provisioned throughput

If you know the primary key of all the items in DynamoDB (some external knowledge like primary is an auto incremented value, is referenced in another DB etc) then you can use BatchGetItem or Query.

So if it is a one off thing then Scan is your only option else you should look into refactoring your application to remove this scenario.