What's a good practice for dealing with DynamoDB batchGetItem's unprocessedKeys

Question

We started using DynamoDB in the past few weeks in our project, both as a cache and as a list of events that occurred in the system (please let's not get into why there are better alternatives to implement this, I've argued that myself before DynamoDB was chosen with no result).

It seems that because of the provisioned throughput limitation, I'm expected to implement in my code a way to retry unprocessed items whenever the limits are exceeded. It makes sense, but it also raises questions regarding batch operations or queries which I can't seem to be able to answer on my own.

I think the BatchPutItem is simple enough to implement. If I get unprocesseditems, I just use an exponential retry and the items will eventually be persisted. I'm doing something like this:

(...)
BatchWriteItemOutcome outcome = dynamoDB.batchWriteItem(new TableWriteItems(tableName).withItemsToPut(items));
processUnprocessed(outcome, 0);
(...)

and

private void processUnprocessed(BatchWriteItemOutcome outcome, int retryNumber) {

    if (MapUtils.isEmpty(outcome.getUnprocessedItems())) {
        return;
    }

    if (retryNumber > maxRetries) {
        log.error(Joiner.on(" ").join("Unable to process", outcome.getUnprocessedItems().size(), "items after", retryNumber, "tries"));
        return;
    }

    long retryTime = (long)Math.pow(retryFactor, retryNumber);
    log.info("Exceeded provisioning throughput. Retrying in " + retryTime);

    try {
        Thread.sleep(retryTime);
    } catch (InterruptedException e) {
        log.error(e.getMessage());
    }

    processUnprocessed(dynamoDB.batchWriteItemUnprocessed(outcome.getUnprocessedItems()), ++retryNumber);
}

Because asynchronous background tasks are populating the DB, this works fine.

For a query or a BatchGetItem, however, it's not that simple. The end-user is waiting for the output of the DynamoDB call. I can't do an exponential retry here, or else the user could be waiting for a really long time. On the other hand, I also cannot not show all the results for the keys I'm asking for.

Does anyone have any suggestion of the right way (I'd settle for a decent way) of handling this? Am I approaching the problem in a wrong way?

I'm using the Amazon JavaSDK btw.

Answer 1

Not really the answer to the question I asked (and I really don't think there is one, feel free do correct me), but I re-worked the way I was thinking about the problem and it actually feels like a well designed solution and not hacky at all. It's pretty obvious when you think about it, but I was completely missing it for a couple of days, so I thought it was worth sharing.

I ended up putting the retry logic on the client for the GetBatchItem only, so that I can show the results that was possible to retrieve straight away. My backend code is 100% free of any retry logic for unprocessed items.

My backend endpoint returns a list of items and a list of unprocessedKeys, as you'd expect.

{
  "items": [{
     "myPartitionKey": "whatever",
     "mySortKey": "whocares",
     "item": "myitem"
   }, (...)],
   "unprocessedKeys": [{
     "pKey": "unprocessed1"
     "sKey": "blah"
   }, (...)]
}

Then it's up to my ReactJs client to partially update the UI based on the data successfully received and to call the same service again only with the missing keys, with some sort of exponential backoff.