AWS messaging system recommendations

Question

I have an application that is connecting to a web socket to get data.

I'd like to store that information by putting it on some sort of queue for multiple consumers to read. I'd like one of those consumers to connect to a time series DB for storage.

I've implemented something in Kafka using InfluxDB, but am looking at AWS alternatives.

• The messages I'm receiving are of a time series / financial nature.
• Latency isn't that important, if they are received by consumers ~1 sec after, that's fine
• The rate of incoming messages is sporadic, with bursts up to ~5 per second
• The order in which they are received is important, and a FIFO protocol is needed.
• The message stream will need to be subscribed to by multiple consumers

I'm a bit lost in all the different products AWS offer. Can anyone recommend a messaging service based on the above requirements? It would be even better if the messaging service connects directly to AWS's timeseries database offering: Timestream without the need for code (a bit like Kafka connect does).

Answer 1

If you are firm with Kafka, try a look at Amazon MSK (managed service for kafka)

The AWS "native technology" way is using SQS with FIFO enabled.

How to do (in short):

Create a SNS topic for your messages and let all your clients send their messages to this topic. In SNS subscribe to a SQS and all your messages will be stored into SQS. You can also have multiple targets subscribed to, like multiple SQS, Kinesis, Lambda, etc.) to deliver your messages into different workflows.

To process your messages, you can then also subscribe to the SQS to get the messages, process and delete them, etc. This depends on your individual need.

I've worked with such a setup (SNS, SQS) and handle about 200-300 messages/minute as normal load without having a large position on the bill in the end of the month.