Akka: testing monitoring\death watch

Question

In my scenario I have 2 actors:

1. watchee (I use TestProbe)
2. watcher (Watcher wrapped into TestActorRef to expose some internal state I track in my test)

Watcher should take some actions when watchee dies.

Here is the complete test case I've written so far:

class TempTest(_system: ActorSystem) extends TestKit(_system) with ImplicitSender with FunSuiteLike with Matchers with BeforeAndAfterAll {

  def this() = this(ActorSystem("TempTest"))

  override def afterAll {
    TestKit.shutdownActorSystem(system)
  }

  class WatcherActor(watchee: ActorRef) extends Actor {

    var state = "initial"
    context.watch(watchee)

    override def receive: Receive = {
      case "start" =>
        state = "start"
      case _: Terminated =>
        state = "terminated"
    }

  }

  test("example") {
    val watchee = TestProbe()
    val watcher = TestActorRef[WatcherActor](Props(new WatcherActor(watchee.ref)))

    assert(watcher.underlyingActor.state === "initial")

    watcher ! "start" // "start" will be sent and handled by watcher synchronously
    assert(watcher.underlyingActor.state === "start")

    system.stop(watchee.ref) // will cause Terminated to be sent and handled asynchronously by watcher
    Thread.sleep(100) // what is the best way to avoid blocking here?
    assert(watcher.underlyingActor.state === "terminated")
  }

}

Now, since all involved actors use CallingThreadDispatcher (all Akka's test helpers gets constructed using props with .withDispatcher(CallingThreadDispatcher.Id)) I can safely assume that when this statement returns:

watcher ! "start"

... the "start" message is already processed by WatchingActor and thus I can make assertions based in the watcher.underlyingActor.state

However, based on my observations, when I stop watchee using system.stop or by sending Kill to it the Terminated message produced as a side effect of watchee death gets executed asynchronously, in another thread.

Not-a-solution is to stop watchee, block thread for some time and verify Watcher state after that, but I'd like to know how to I do this the right way (i.e. how to be sure that after killing actor it's watcher received and processed Terminated message signaling it's death)?

Answer 1

EDIT: After discussion and testing with OP, we have found out that sending PoisonPill as a mean of terminating the watched actor achieves the desired behavior, as the Terminated from PPill are processed synchronously while those from stop or kill are processed asynchronously.

While we are not sure about the reason, our best bet is this is due to the fact that killing an actor raises an exception while PPilling it does not.

Apparently, this has nothing to do with using the gracefulStop pattern as per my original suggestion, which I'll leave reported below.

In summary, the solution to OP's problem was just to terminate the watched actor sending a PPill instead that a Kill message or by doing system.stop.

---

Old answer begins here:

I may be going to suggest something a bit unrelated, but I feel it may apply.

If I understood correctly what you want to do is basically terminate an actor synchronously, i.e. do something that returns only once the actor is officially death and its death has been recorded (in your case by the watcher).

In general, death notification, as well as most else in akka, is async. Nonetheless, it's possible to obtain synchronous death confirmation using the gracefulStop pattern (akka.pattern.gracefulStop).

To do so, the code should be something similar to:

val timeout = 5.seconds
val killResultFuture = gracefulStop(victimRef, timeout, PoisonPill)
Await.result(killResultFuture, timeout)

What this does is sending a PoisonPill to the victim (note: you can use a custom message), who will reply with a future that is completed once the victim dies. Using Await.result you are guaranteed to be synchronous.

Unfortunately, this is only usable if a) you are actively killing the victim and you don't instead want to respond to an external cause of death b) You can use timeouts and blocking in your code. But maybe you can adapt this pattern to your situation.