I am starting to use WebJobs to handle some background tasks and I am having issues regarding error handling and retrying. It appears that if any of my functions are async, then the function always reports success in the dashboard even though I throw an exception within the function.
Consider the following simple example ->
public static void AlwaysFail([QueueTrigger("alwaysfail")] string message)
{
throw new Exception("Induced Error");
}
The code above behaves as I expect it to. The message is popped off the queue, exception is thrown, fail reported in the WebJobs dashboard, message is requeued. This occurs 5 times, and then the message is stored in the poison queue.
But if I attempt a similar function but make it async like below ->
public async static void AlwaysFail([QueueTrigger("alwaysfail")] string message)
{
throw new Exception("Induced Error");
}
The results are not as expected. The message is popped off the queue, exception is thrown, but the dashboard reports Success and nothing is retried. Furthermore, in these cases, it appears that although the dashboard is reporting Success, the exception is crashing the process with an out of this world exit code, waiting 60 seconds, and restarting as shown in the following log ->
[11/11/2014 06:13:35 > 0dc4f9: INFO] Executing: 'Functions.AlwaysFail' because New queue message detected on 'alwaysfail'. [11/11/2014 06:13:36 > 0dc4f9: ERR ] [11/11/2014 06:13:36 > 0dc4f9: ERR ] Unhandled Exception: System.InvalidOperationException: Induced Error [11/11/2014 06:13:36 > 0dc4f9: ERR ] at Plmtc.BackgroundWorker.Functions.d__2.MoveNext() [11/11/2014 06:13:36 > 0dc4f9: ERR ] --- End of stack trace from previous location where exception was thrown --- [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Runtime.CompilerServices.AsyncMethodBuilderCore.b__5(Object state) [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(Object state) [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Threading.ThreadPoolWorkQueue.Dispatch() [11/11/2014 06:13:36 > 0dc4f9: ERR ] at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback() [11/11/2014 06:13:36 > 0dc4f9: SYS ERR ] Job failed due to exit code -532462766 [11/11/2014 06:13:36 > 0dc4f9: SYS INFO] Process went down, waiting for 60 seconds [11/11/2014 06:13:36 > 0dc4f9: SYS INFO] Status changed to PendingRestart
The only way I can get an async function to fail in the dashboard to to write one that takes a POCO class as a parameter but only include a simple string in the message. In these cases, the failure is occurring during the message/parameter binding before any of the function's code is reached.
Anyone out there successfully using async functions to respond to queue triggers without these issues?
Because the method is async void, the WebJobs SDK is unable to await the method to its completion. Change it to return a Task...
public async static Task AlwaysFail([QueueTrigger("alwaysfail")] string message)
{
throw new Exception("Induced Error");
}
See this post for more details.