Search code examples
c#.netakka.net

How to clear the mailbox when restarting an actor in Akka.NET


I'm developing a small actor system, where when something goes wrong in the actor hierarchy, I want all actors to be shutdown and the top level one be restarted, so everything starts from a clean state.

I can do it rather easily, implementing the supervision strategy in the intermediate actors to escalate the error, then having the top level actor restart. However, as described in this video, a restart doesn't clear the actors mailbox, which I'd like to do to really ensure everything starting over clean.

So far, I haven't found any documentation on how to clear the mailbox, so not sure it's even possible.

However, as I don't want to just restart, but have some delay, I looked into BackoffSupervisor. From what I understand, when an actor is restarted by the BackoffSupervisor, it's not actually restarted, but stopped during the backoff period, then started afresh, which means its mailbox is actually cleared, as I want. This solution works for me, but I'd like to be sure this behavior is like this on purpose, and I can rely on it for this use case.

Probably not super relevant, but for reference, just in case, this is how I'm creating this supervisor:

return BackoffSupervisor.Props(
    Backoff.OnFailure(
        rootActorProps,
        childName: nameof(RootActor),
        minBackoff: TimeSpan.FromSeconds(1),
        maxBackoff: TimeSpan.FromMinutes(5),
        randomFactor: 0.2,
        -1));

So, in summary, my doubts are:

  • Is it possible to somehow clear an actor's mailbox?
  • Is using BackoffSupervisor a valid approach to get the mailbox, and everything else restarted in a clean state

Solution

  • Here's a LINQPad script I wrote - please ignore the ActorSytem termination stuff; I just needed that to make sure the program stopped running ;)

    async Task Main()
    {
        // Initialize Actor System
        var system = ActorSystem.Create("MySystem");
    
        // Create Props for the EchoActor
        var props = Props.Create<EchoActor>();
    
        // Create the EchoActor
        var echoActor = system.ActorOf(props, "echoActor");
    
        // Send 10 messages to the EchoActor
        for (int i = 0; i < 10; i++)
        {
            echoActor.Tell($"Message {i + 1}");
        }
    
        // Allow some time for messages to process before shutting down
        await system.WhenTerminated;
    }
    
    // Define the EchoActor class
    public class EchoActor : ReceiveActor
    {
        public EchoActor()
        {
            NormalBehavior();       
        }
    
        private void NormalBehavior()
        {
            ReceiveAnyAsync(async message =>
            {
                await Task.Delay(100);
                Console.WriteLine($"Received and echoing back: {message}");
                throw new ApplicationException("restart");
            });
        }
        
        private void RecoverBehavior(){
            Receive<RecoverMarker>(_ =>{
                Become(NormalBehavior);
                Context.System.Terminate();
            });
            
            ReceiveAny(_ =>{
                // discard
                Console.WriteLine($"Discarding: {_}");
            });
        }
    
        private class RecoverMarker{}
        
        protected override void PreRestart(Exception ex, object message){
            Self.Tell(new RecoverMarker());
        }
        
        protected override void PostRestart(Exception ex){
            Become(RecoverBehavior);
        }
    }
    

    How this sample works: it uses behavior switching to move the actor into "discard" mode immediately after it is restarted, and the "restart marker" is pushed into the actor's mailbox earlier when the old instance of the actor is being destroyed. The actor will discard all older messages up until it hits that RestartMarker.

    This is a generalized approach to this problem and it works without having to touch internal Mailbox APIs et al.