Search code examples
notificationssql-server-2016job-queue

Tfs 2017 Update 2 - "WorkItem user delivery" job stuck - no email/soap notifications


I have a job "WorkItem User Delivery" that seems to be stuck in TFS 2017 Update 2 (SQL Server 2016). It's been running for 4 days and is causing issues with the other mail/soap jobs and email alerts are very sporadic: Image of JobQueue in TFS OI

I've located the job in [Tfs_Configuration].[dbo].[tbl_JobQueue] with a JobID of 631F49B3-46E1-42EC-8FFF-081BD176C18A. I've already restarted TFSJobAgent with no luck. Is it safe to delete this row from the table and will that be effective in removing the job?


Solution

  • We recently added two new application tiers to our production infrastructure as we get ready to migrate to a new On-Premises data center. These two new application tiers were not added to our load balanced VIP so no load was actually getting to them (except when I hit them directly).

    These new machines didn't have firewall access to the machine that is listening for SOAP calls in the current prod data center (and possibly not even to the mailhost). I believe TFS offloaded these notification/soap jobs to one of these new machines because as soon as I turned off TFSJobAgent on all four machines and turned it back on on only the two original machines did everything start working again.