Background...
I have a Windows Service with many components in it that all run on schedules. Each component processes files but performs specific tasks to the files based on the current status they are at in our system. These components currently create Threads (System.Threading.Thread) for each file being processed, within each component in the service.
Component 1 might be pulling files down from an FTP site.
Component 2 might be unzipping a file to the hard drive.
Component 3 might be decrypting files on the hard drive.
Component 4 might be copying or moving files from one place to another.
Currently, each component kicks off its component specific task in a new Thread, per file being processed. This has worked out well for us, but as the system and company grows, it's becoming more and more difficult to manage. I am looking into the ThreadPool (System.Threading.ThreadPool) for both easier thread management and better resource management overall.
Here's a simplified current design...
'send a message that the component task has begun
Comms.ComponentStatus(ComponentID, Running)
For Each f As File
Dim t As New Thread(AddressOf processFile)
t.Start(f)
lst.Add(t)
Next
'some list checking to see if all threads in list are done
Do Until lst has no active threads
Loop
'all threads complete so the component task is complete
Comms.ComponentStatus(ComponentID, Complete)
My dilemma...
I created a dashboard that receives real-time messages (.Net Remoting) about each component and task being performed. Trace info, exception info and most importantly the Start and End of a component's task overall. In my current design, I message that a task has begun, create threads for each file to be processed and keep track of the threads created. I look at all threads created for the task and when they are all complete, I message that the task has completed. This works very well for us. With a ThreadPool design, all of my components will be pulling threads from a process-wide thread pool, allowing the system to manage them, but not allowing me to know which threads are being used for which tasks within each component, therefore not allowing me to know when a component's task is complete.
A quick look into .Net's ThreadPool does not show me that I can determine which active threads in the pool are performing which tasks. Does anyone have a solution or suggestion? Thanks in advance.
'this only returns a bool telling me the requested task will be performed
ThreadPool.QueueUserWorkItem
'this only returns to me a number of threads available for queue
ThreadPool.GetAvailableThreads()
What I decided to do is create a Class that holds onto information about the task being performed including the Component's ID, the Task's ID, and a Thread Count. Whenever the task runs, it creates an instance of this object at the component level (Class level). This allows me to isolate counts of threads at the component level. When it queues threads in the ThreadPool, it increments the thread counter in the object. When all threads have been queued it sits and waits for the thread counter to return to 0, therefore marking the task as complete. The thread counter gets decremented each time the delegate for the thread has finished processing (the thread is complete). I put a catchall timespan to get out of the thread count loop in case something unforeseen happens. A quick code and test showed me conceptually it does work. I will continue to code and test and if any findings occur that result in changes, I will post them here.
Here's my initial outline of the thread tracking object.
Public Class TaskTracker : Implements IDisposable
Public ReadOnly Property ComponentID As Integer = 0
Public ReadOnly Property TaskUID As Guid = Guid.Empty
Public ReadOnly Property ThreadCount As Integer = 0
''' <summary>
''' Create a new instance of the TaskTracker object.
''' </summary>
''' <param name="ComponentID">The ID of the Component this object belongs to.</param>
''' <param name="TaskUID">The UID of the Task in question.</param>
Public Sub New(ComponentID As Integer, TaskUID As Guid)
Try
_ComponentID = ComponentID
_TaskUID = TaskUID
_ThreadCount = 0
Catch ex As Exception
Log.Save(Log.Types.Error, ComponentID, TaskUID, ex.Message, ex.StackTrace)
End Try
End Sub
''' <summary>
''' Increment the internal thread count property by the amount in the value provided.
''' </summary>
''' <param name="Value">The amount to increment the thread count by.</param>
Public Sub IncrementThreadCount(Optional Value As Integer = 1)
Try
_ThreadCount += Value
Catch ex As Exception
Log.Save(Log.Types.Error, ComponentID, TaskUID, ex.Message, ex.StackTrace)
End Try
End Sub
''' <summary>
''' Decrement the internal thread count property by the amount in the value provided.
''' </summary>
''' <param name="Value">The amount to decrement the thread count by.</param>
Public Sub DecrementThreadCount(Optional Value As Integer = 1)
Try
If _ThreadCount > 0 Then
_ThreadCount -= Value
Else
_ThreadCount = 0
End If
Catch ex As Exception
Log.Save(Log.Types.Error, ComponentID, TaskUID, ex.Message, ex.StackTrace)
End Try
End Sub
Private disposedValue As Boolean
Protected Overridable Sub Dispose(disposing As Boolean)
If Not disposedValue Then
If disposing Then
End If
_ComponentID = 0
_TaskUID = Guid.Empty
_ThreadCount = 0
End If
disposedValue = True
End Sub
Public Sub Dispose() Implements IDisposable.Dispose
Dispose(True)
End Sub
End Class
Here's its (abbreviated) implementation...
Private Shared taskTrack As TaskTracker = Nothing
Public Shared Function Start() As ResultPackage
Try
TaskUID = Guid.NewGuid()
taskTrack = New TaskTracker(ComponentID, TaskUID)
'let any listeners know the task has started
Comms.ComponentStatus(ComponentID, True)
'mark the start time of the total task
compStart = Now
Log.Save(Log.Types.Trace, ComponentID, TaskUID, _ClassName & " Started", "Successful start of the " & _ClassName & " component.")
For Each cli As ClientMaster In ClientMaster.GetList(True)
'inner try/catch so that we can continue to process clients even if one errors
Try
ThreadPool.QueueUserWorkItem(AddressOf processClient, cli)
Log.Save(Log.Types.Trace, ComponentID, TaskUID, "Client Thread Queued", "Thread queued for Client [" & cli.ClientName & "].")
taskTrack.IncrementThreadCount()
Catch ex As Exception
Log.Save(Log.Types.Error, ComponentID, TaskUID, ex.Message, ex.StackTrace)
End Try
Next
Do Until taskTrack.ThreadCount = 0 'or some timespan has been reached for a catchall
Thread.Sleep(500)
Loop
Comms.ComponentStatus(ComponentID, False)
'mark the end time of the total task
compEnd = Now
Log.Save(Log.Types.Trace, ComponentID, TaskUID, _ClassName, "Successful end of the " & _ClassName & " component.")
Log.Save(Log.Types.Trace, ComponentID, TaskUID, _ClassName & " Task Time", _ClassName & " task took " & FriendlyTimeSpan(DateDiff(DateInterval.Second, compStart, compEnd)) & " to complete.")
Comms.ComponentMessage(ComponentID, "Task Duration: " & FriendlyTimeSpan(DateDiff(DateInterval.Second, compStart, compEnd)))
Catch ex As Exception
resPack.Result = ResultPackage.Results.Fail
resPack.Alerts.Add(New ResultPackage.Alert("Exception in Start()", ex))
Log.Save(Log.Types.Error, ComponentID, TaskUID, ex.Message, ex.StackTrace)
Throw
End Try
Return resPack
End Function
Private Shared Sub processClient(Client As ClientMaster)
'do work
'decrease the thread counter since this task is complete
taskTrack.DecrementThreadCount()
End Sub