I have to implement a process which polls messages from a queue and forwards the data to a http endpoint. Normally I think the best approach for this would be to use Akka streams (with akka http), because it handles backpressure etc. for you. The problem is that I have to create multiple of these pipelines on demand during runtime (based on a http call) with different configurations. Another requirement is that it should be possible to dynamically stop one of those pipelines during runtime and start it with another configuration.
I'm currently not sure if it is really possible to dynamically spin up new parallel akka streams pipelines during runtime (and also remove and restart some of them). For me it currently sounds better to use akka actors for this (based on a queue consumer actor and a akka http actor and I could create a pair dynamically during runtime), but to use them I also have to implement backpressure etc. manually.
Do you think it would be possible to create this via akka streams or would it be a hassle?
Thanks in advance!
You cant reroute the stream once its meterialized and running. But since you configure streams programmatically it is very handy to dynamically define new streams and run them. Everything depends on certain use case but I'd go for single coordinator actor which receives queue messages and dispatch them to the given stream. The same coordinator would receive configuration change requests so it configure new streams, removes old ones etc.
Dont go for manually wiring actors. Its hard to follow and maintain.