I understand that Akka actors should not block in order to stay reactive to messages, but how do I structure my service where I want to monitor a process running for an indefinite period of time?
For example, we are using the Amazon Kinesis Connector library. You create a connector with a given configuration, which inherits from Runnable, and then call the Run() method. The connector simply runs indefinitely, pulling data from Kinesis, and writing it to Amazon S3. In fact, if the runnable returns, then that is an error, and it needs to be restarted.
Approach (1) would be to simply create a child actor for each Kinesis Connector running, and if the Run() method returns, you throw an exception, the Supervising Actor notices the exception and restarts the child actor. One connector per child actor per thread.
Approach (2) would be for the child actor to wrap the Kinesis Connector in a Future, and if the future returns, the actor would restart the Connector in another Future. Conceivably a single actor could manage multiple Connectors, but does this mean each Future is executing in a separate thread?
Which approach would be most in line with the philosophy of Akka, or is there some other approach people recommend? In general, I want to catch any problems with any Connector, and restart it. In total there would not be more than a half dozen Connectors running in parallel.
I would take approach 1. It should be noted though that actors do not have a dedicated thread by default but they share a thread pool (the so called dispatcher, see: http://doc.akka.io/docs/akka/2.3.6/scala/dispatchers.html). This means that blocking is inherently dangerous because it exhausts the threads of the pool not letting other non-blocked actors to run (since the blocked actors do not put the thread back into the pool). Therefore you should separate blocking calls into a fixed size pool of dedicated actors, and you should assign these actors a PinnedDispatcher. This latter step ensures that these actors do not interfere with each other (they each have a dedicated thread) and ensures that these actors do not interfere with the rest of the system (all of the other actors will run on another dispatchers, usually on default-dispatcher). Be sure though to limit the number of actors running on the PinnedDispatcher since the number of used threads will grow with the number of actors on that dispatcher.