Handle background processing once

Question

I have a Blazor Interactive Server app running on Azure App Services. It scales out/in with a minimum of 2 instances and it got up to 5 once.

In the app I have a BackgroundService that I use to send emails, synchronize with an external service, mark events closed when they have hit their end date, etc. It runs once every 5 minutes and when I signal it (changed something I want to get synchronized ASAP).

If there was one instance then this is a piece of cake. I read all the pending items from the DB and act on them and/or update the database record.

But with 2+ instances there's the issue of sending 2 emails. I can set things up so I get a DbUpdateConcurrencyException when I've sent it twice but by the time I get that it's too late - I've sent 2 emails.

And with Azure spinning up new instances and shutting down old ones to run Windows update, etc. I can't find any way to identify one instance to be the only one running. Plus I need to signal the worker in every instance.

So, is there some way I can create a semaphore or something in the database. Where it then is the only background thread allowed to run for 2 minutes (time limit in case Azure then shuts down that server)? Or some other approach?

Because all the work this worker does is based off the database, if any one instance is processing the background items, it will get all items wanted by all instances. So there's no need to signal the other instances when one instance is done processing.

Update: Based on the comments below, let me add a bit.

1. Leadership election - yes that's what I am leaning toward. Using a DB table for this. But it can't be a permanent decision because server instances get closed down. In addition, I need any instance able to kick off the process. So I think maybe more like what goes on with network communication where each, via the DB, tries to grab a semaphore when it wants to run. And if it wins, runs. But the big question then is, how specifically to do this.
2. Making the DB a message queue for some of this works. That adds complexity though where it becomes try to take assignment of an item. And if an item was grabbed 5 minutes ago, but is still held, then assume that instance died. However a big problem with this is the most important set of items must be processed in order. Item 1 causes an object to be added to the database that item 2 needs. So more complex, tricky re-try logic, and possible stalling of the queue. So not wild on this approach.
3. I am considering having a function do this. But that adds the problem of signaling it (I assume Azure has a way to do this). Bigger problem, I now have a single point of failure. Or I have 2 functions, in different data centers, and then I'm back with this original problem.

That's why this is so tricky. For some of the key items they need to be processed in order, I want to have multiple instances available to run, and I need the ability for all app server instances to be able to kick off the processing.

On the good news side, if an instance is signaled and another instance is running, that works. The second instance can do nothing.

Answer 1

There is a beautiful library that does exactly this - DistributedLock. Works great.