How to push persistent events from a central server to others?

Question

I have a system that consists of one central server, many mobile clients and many worker server. Each worker server has its own database and may be on the customer infrastructure (when he purchases the on premise installation).

In my current design, mobile clients send updates to the central server, which updates its database. The worker servers periodically pull the central to get updated information. This "pull model" creates a lot of requests and is still not suficient, because workers often use outdated information.

I want a "push model", where the central server can "post" updates to "somewhere", which persist the last version of the data. Then workers can "subscribe" to this "somewhere" and be always up-to-date.

The main problems are:

1. A worker server may be offline when an update happen. When it come back online, it should receive the updates it lost.
2. A new worker server may be created and need to get the updated data, even the data that was posted before it exists.

A bonus point:

3. Not needing to manage this "somewhere" myself. My application is deployed at AWS, so if there's any combination of services I can use to achieve that would be great. Everything I found has limited time data retention.

Answer 1

The problems with a push model are:

• If clients are offline, the central system would need a retry method, which would generate many more requests than a push model
• The clients might be behind firewalls, so cannot receive the message
• It is not scalable

A pull model is much more efficient:

• Clients should retrieve the latest data when the start, and also at regular intervals
• New clients simply connect to the central server -- no need to update the central server with a list of clients (depending upon your security needs)
• It is much more scalable

There are several options for serving traffic to pull requests:

• Via an API call, powered by AWS API Gateway. You would then need an AWS Lambda function or a web server to handle the request.
• Directly from DynamoDB (but the clients would require access credentials)
• From an Amazon S3 bucket

Using an S3 bucket has many advantages: Highly scalable, a good range of security options (public; via credentials; via pre-signed URLs), no servers required.

Simply put the data in an S3 bucket and have the clients "pull" the data. You could have one set of files for "every" client, and a specific file for each individual client, thereby enabling individual configuration. Just think of S3 as a very large key-value datastore.