Synchronizing multiple threads in .net 4

Question

Background: I'm working on an application that already has a thread system designed. It's far from optimal, but I can't rework it at this time. It does not use any of the newer threading constructs from .net, just basic Thread objects, and objects wrapping the thread processing logic.

The general set up of it is: (there are two groups, or categories of threads right now)

• Main application thread (statthread main) which spawns worker threads.

• Worker threads. Each one has a Thread object, and a worker object which does all the processing / handles the boundary between threads, etc. Each thread runs jobs, and each job has a JobTypeID.

I need to introduce a third type of thread, one for controlling the workers. These control messages will come from a a wcf web service (so this thread is handled implicitly).

The control messages are: { Pause/Resume, List-of-IDs }

My goal:

I'm trying to figure out what the best way to synchronize these threads such that if a thread is processing a job, and a message comes in saying to pause all jobs of this JobTypeID, it should block until a resume is sent (for that ID). The catch here is that at the time that the message is sent, no relevant jobs might be processing, so no immediate action will need to be taken, and I also don't have a list of worker objects, so I can't simply iterate over each worker and do if-matches-then-pause.

Actual question (generalized) What would you guys recommend I do to synchronize a set of worker threads, a spawner/manager thread, and a set of control threads?

things i've tried

One approach would be to store a collection of ManualResetEvent objects, one per JobTypeID, and to signal and wait them according to the incoming messages. What do you think about this approach? I can't find any information about best practices or memory/processing costs for having 100+ wait handles in a process.

Another approach would be to have a single object on which all the threads would wait, and a synchronized collection of JobTypeIDs that should wait. I've had some problems with this approach. Using a ManualResetEvent means that if I resume one job id, but others are waiting, I have to do set();reset();, and this causes some race conditions (even if I try to do WaitHandle.SignalAndWait(x,x)) Finally, I came up with a solution using Monitor.PulseAll() which works.

I could also use a monitor with lots of lock objects - this seems like it would be more lightweight than lots of waithandles.

Also, sorry for the long question and thanks for reading!

Answer 1

Create a ConcurrentDictionary<JobTypeId, ManualResetEvent>. Normally, there will be no entry in the dictionary for a JobTypeId. The only time such an entry will exist is when that job type is supposed to be blocked.

When that job type is supposed to be blocked, create a ManualResetEvent that is unsignaled, and add it to the dictionary, with the key being the JobTypeId to be blocked.

The worker threads, then, have a loop that looks like this:

while (still_have_work_to_do)
{
    ManualResetEvent mevent;
    if (PauseDictionary.TryGetValue(myJobTypeId, out mevent))
    {
        // wait until the event is signaled.
        mevent.WaitOne();
    }

    // Do more processing.
}

The spawner thread could also query the dictionary to see if it can spawn a particular type of worker. If it can't create the worker, then it either discards that job or re-queues it to be checked at a later time. How often you poll is up to you.

There's no particular problem with having hundreds of WaitHandle objects in a single process. A bunch of Monitor objects might take fewer system resources, but remember that a Monitor (or a lock) is really a mutual exclusion device. You can do silly things to make it act like a WaitHandle in some respects, but the techniques are non-obvious, which will lead to hard to understand and fragile code.