I have a quick question. Is the KCL able to consume from multiple streams? Should you ever set up multiple streams for your application, or is a individual stream supposed to be tied with an individual application? My particular use case is that I need to consume data being produced from the backend and also from the frontend. One of these produces data at much greater rates than the other, and for that reason think they should produce into separate streams for processing. Is there a way to consume both streams from the same KCL process or do I need to set up two? Thanks for your help!
KCL is an open source project that you can modify to consume events from multiple streams, but this is not recommended. It is better to keep things simpler.
If you have 2 different event streams, you better have 2 different kinesis streams, one for each. This allows you to scale each stream independently as each has a different rate and possibly different peaks.
If you need to share information between the streams, you can use share state variables between them, using some DB such as DynamoDB or Redis.
Please note that if you have a set of servers that are sending out these events, you should expect that some of the events of the back end, might be processed before the events from the front end. The KCL (or Lambda) code that you will have to process these events, can have different processing rates, different failure points and other out-of-synch events. Take note of such potential dependencies and exceptions.