How to make active services highly available?

Question

I know that with Network Load Balancing and Failover Clusteringwe can make passive services highly available. But what about active apps?

Example: One of my apps retrieves some content from a external resource in a fixed interval. I have imagined the following scenarios:

1. Run it in a single machine. Problem: if this instance falls, the content won't be retrieved
2. Run it in each machine of the cluster. Problem: the content will be retrieved multiple times
3. Have it in each machine of the cluster, but run it only in one of them. Each instance will have to check some sort of common resource to decide whether it its turn to do the task or not.

When I was thinking about the solution #3 I have wondered what should be the common resource. I have thought of creating a table in the database, where we could use it to get a global lock.

Is this the best solution? How does people usually do this?

By the way it's a C# .NET WCF app running on Windows Server 2008

Answer 1

For such problems they have invented message queues. Imagine the case when your clustered applications all listen to a message queue (clustered itself :-)). At some point in time one instance gets your initial command to download your external resource. If successful, your instance flushes the message and instead it posts another one for a later execution time that's equal to 'the run time' + 'interval'. But in case the instance dies during processing, that's not a problem. The message is rolled back in the queue (after timeout) and some other instance can pick it up. A bit of transactions, a bit of message queues

I am on the Java EE side of the world so can help you with coding details