Using Realm with Amazon DynamoDB

Question

My app is offline-first, and therefore Realm has been wonderful for persisting and accessing data. I love it. However, I also want to store the user data in the cloud (for backup but also in case I add web support later). I know that's exactly what Realm Object Server is for, but I think I'd prefer to use DynamoDB for the following reasons:

1) I've already invested in DynamoDB and Amazon's authentication (Cognito).

2) I like that Realm is effectively a relational database because from the client I do need to run complex queries. However, on the backend I mostly just want to backup all the data in a way that I can easily access it and manipulate if needed through Lambda functions). I'm totally fine with a NoSQL solution for this, and my understanding is that DynamoDB is a cost effective database with horizontal scaling, which is appealing to me. If I wanted to access the data in this way with Realm Object Server, my understanding is it would cost at least 1,500 a month.

3) No offense to the Realm team, but I got screwed by Parse shutting down and so I'd like to use something that I can trust will be around for 5+ years as my backend.

Anyway, with that out of the way, here's how I'm currently making this work:

1) Whenever I create or edit a Realm object, I have logic that will map that change into my DynamoDB schema (which is made up of far fewer tables than Realm).

2) I call these updates UpdateTasks and I queue them up and merge them as needed (if you, for instance, changed the same property more than once).

3) I go through the queue and pass chunks of UpdateTasks to a Lambda function I wrote that will iterate through the updates and perform the necessary put or update commands to DynamoDB.

4) I have retry logic in place in case you're offline or a request fails

5) Assuming everything is synced properly if you got a new phone and signed in, I have a separate Lambda function that will fetch all of the user's data and populate the Realm file just as it was before.

Like I said, all of this is working right now, but it feels fragile, and I can't help but feel like I'm going about this the wrong way. Plus it doesn't support two-way syncing or real-time communication if I wanted to add some social features

So my question is if this is a reasonable approach to making Realm sync with DynamoDB or if there's a better/more robust way? Also if I should reconsider using Realm Object Server or something else instead of DynamoDB, I'd be interested to hear why.

It is a big decision for me, so I'd appreciate all the help I can get! Thanks

Answer 1

Disclaimer: I'm the lead on the ROS, so I'm obviously biased. I'm posting this not as an answer, but as general comments that wouldn't fit in the comment format.

I like that Realm is effectively a relational database because from the client I do need to run complex queries.

You're going to anger the gods by calling Realm a relational database. We support queries, and links, and all that, but we're very, very different from a relational database ;).

Also if I should reconsider using Realm Object Server or something else instead of DynamoDB, I'd be interested to hear why.

You are missing out on a number of very important features offered by the Realm Mobile Platform by reimplementing sync yourself.

Mainly, we have a few thousand lines of code to handle conflict-free merge resolution, in the case where you have multiple clients writing data to the same Realm, while all offline. In addition:

• We handle the network for you. You don't have to worry or care about setting up connections, the protocol between the client and the server, nor whether things are getting serialised correctly.
• Our protocol is also extremely lightweight, because we are able to transfer only the transaction log from the Realm database; not serialised objects or anything like that.
• Our sync is extremely fast. You've probably seen the draw demo video when we launched the RMP. Because our sync is integrated directly into the Realm, we are able to squeeze extra performance through. We have built a number of chat apps and what not internally, and people are continuously surprised at the speed at which messages are transferred.
• You work with the same SDK and API on the server and client (although a different language).
• We are continuously adding features that you won't have to build manually (such as the often requested partial replication and object-level permissions).

On the other hand:

The system you have built very closely resembles the techniques we employ to send orders/requests to the server backend. Typically, we create FooRequest objects on the client, those get synced, and then the event handler picks those up, handles them, and creates a FooResponse object which gets synced back to the client.

Overall, I'm very impressed that you've built all of this on top of vanilla-Realm. If we get the opportunity one day, I'd love to have a look at your whole stack.

Finally, we are working on our pricing, because your feedback regarding the price point isn't unique (but it is very helpful!).