Search records by value or conditional value in Nosql with 100 millions records

Question

We are looking NoSQL database where we can store more than 100 million records with many fields in Value like sets in Redis.

And database should be searchable with value. We checked Redis but it not supporting any option to search by value. because we have millions of records and we update some fields of records and then take a bunch of records which not updated at a specific time.

So, run a query on all records and then check which records not update from specific time take more time. Because in this solutions we are updating 100-200 records per minute and then take bunch record based on value.

So, Redis will not work here. We have the option to store into MongoDB but we are looking key-value database which supports search by value kind of features.

{ 
    "_id" : ObjectId("5ac72e522188c962d024d0cd"), 
    "itemId" : 11.0, 
    "url" : "http://www.testurl.com", 
    "failed" : 0.0, 
    "proxyProvider" : "Test", 
    "isLocked" : false, 
    "syncDurationInMinute" : 60.0, 
    "lastUpdatedTimeUTC" : "", 
    "nextUpdateTimeUTC" : "", 
    "targetCountry" : "US", 
    "requestContentType" : "JSON", 
    "group" : "US"
}

Answer 1

In Aerospike, you can use predicate filtering to find records that have not been updated since a point in time, and return only the metadata of that record, which includes the record digest (its unique identifier). You can process the matched digests and do whatever update you need to do. This type of predicate filter is very fast because it only has to look at the primary index entry, which is kept in memory. See the examples in the Java client's repo.

You would not need to use a secondary index here, because you want to scan all the records in a namespace (or set of that namespace) and just check the 'last-update-time' piece of metadata of each record. Since you'll be returning just the record's digest (unique ID) and not any of its actual data, this scan will never need to read anything from SSD. It'll be very fast and lightweight on the results (again, only metadata is sent back). In the client you'll iterate the result set, build a list of IDs and then act on those with a subsequent write.