Search code examples
azureazure-webjobsazure-webjobssdk

Azure webjob - QueueTrigger stops triggering

I am running an azure webjobs SDK console application (continuous) with the recommended setup:

public static void ProcessQueueMessage([QueueTrigger("logqueue")] string logMessage, TextWriter logger)

The azure queue I am running against has ~6000 messages in it and I am running the web-job locally, as a console application.

The problem I'm having is that the processing randomly stops after processing between zero and ~30 messages. The console stays open, but no more console messages are displayed.

For example, it might just process 2 messages:

Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New queue message detected on 'QueueName'.'
Executed: 'Functions.ProcessQueueMessage' (Succeeded)

And then, nothing. There doesn't seem to be anything wrong with my internet connection and I can't trace the issues down to any particular messages.

Has anyone else had issues with this SDK?

Update:

I made sure that I was using the right versions of all of the dependencies by removing the nuget packages and then re-running install-package Microsoft.Axure.Webjobs. I am now using webjobs version 1.1.0 which has pulled in version 4.3 of azure storage.

As recommended by Matthew, I have pulled down the source code for azure webjobs to determine where the process is freezing up. Once the freez-up occurs, I pause execution and checked the running threads for what I believe is the culprit within Microsoft.Azure.WebJobs.Host.CompositeTraceWriter

    protected virtual void InvokeTextWriter(TraceEvent traceEvent)
    {
        if (_innerTextWriter != null)
        {
            string message = traceEvent.Message;
            if (!string.IsNullOrEmpty(message) &&
                 message.EndsWith("\r\n", StringComparison.OrdinalIgnoreCase))
            {
                // remove any terminating return+line feed, since we're
                // calling WriteLine below
                message = message.Substring(0, message.Length - 2);
            }

            _innerTextWriter.WriteLine(message);
            if (traceEvent.Exception != null)
            {
                _innerTextWriter.WriteLine(traceEvent.Exception.ToDetails());
            }
        }
    }

The line it freezes on is line 66 : _innerTextWriter.WriteLine(message);

_innerTextWriter is an instance of System.IO.TextWriter.SyncTextWriter

Is it possible there is some deadlock issue with this class or the way it is being used?

Some notes:

  • I am running in the debugger, so in this case I believe the textwriter is forwarding to the console internally
  • I have my batchsize set to 1 via config.Queues.BatchSize = 1;, not sure if that could matter

I'm currently working on setting up an environment on another computer so that I can see if it is reproducible somewhere other than this machine (surface book).

Update

The issue was me not understanding how the new windows 10 command prompt works. Any time you click on the command window, it goes into "select" mode which completely pauses execution of the process.

Basically: https://superuser.com/questions/419717/windows-command-prompt-freezing-randomly?newreg=ece53f5584254346be68f85d1fd2f18d

You can tell it is in this state because it will prefix the window title with the word "Select":

frozen command window

You have to press enter or click again to get it going once again.

So, two final comments:

1) What an incredibly confusing and un-intuitive behavior for a command window!

2) I hope some admin will come take pity on the shame I have brought upon myself and my family by deleting this question.

To get rid of this strange behavior, you can disable QuickEdit mode:

disable QuickEdit Mode

Solution

  • Strange. When it is in this stuck state, can you try adding a new queue message to the queue and see if that triggers? Are you sure your function isn't hanging internally? What version of the SDK are you using? You might also try upgrading to v1.1.0 which we just released last week. If there are really a bunch of messages in the queue waiting to be processed, I can't think of anything that would cause this. The queue listener in the SDK should chug along, reading batches of messages in parallel and dispatching them to your function. Have you changed any of the JobHostConfiguration.Queues configuration knobs? You haven't force updated the version of the Azure SDK have you to something higher than the WebJobs SDK supports?

    Another option if you can't figure this out might be to clone the SDK, build it and debug it locally. The repo is here. The main queue processing loop is here.