My job executes an arbitrary number of times even though only scheduled once

Question

• Hangfire 1.6.17
• SQL Server 2016 (13.0.4206.0)

I have a C# console application that uses OWIN to host a Hangfire server as follows:

public class Startup
{
    public void Configuration(IAppBuilder aAppBuilder)
    {
        //How we want to use Hangfire
        GlobalConfiguration.Configuration
                           .UseSqlServerStorage("HangfireConnectionString")
                           .UseConsole(new ConsoleOptions
                                       {
                                           BackgroundColor = "#0d3163",
                                           TextColor = "#ffffff",
                                           TimestampColor = "#00aad7"
                                       });

        //Read the settings file
        SettingsIni iniSettings = new SettingsIni(GeneralHelper.GetAppPath() + "settings.ini");

        //Dashboard
        aAppBuilder.UseHangfireDashboard(iniSettings.HttpServer.Path, new DashboardOptions
                                                                      {
                                                                          AppPath = "/",
                                                                          Authorization = new[]
                                                                                          {
                                                                                              new HangfireAuthorizationFilter()
                                                                                          }
                                                                      });

        //Server
        BackgroundJobServerOptions serverOptions = new BackgroundJobServerOptions
                                                   {
                                                       Queues = GeneralHelper.GetQueueNames(iniSettings).ToArray()
                                                   };

        aAppBuilder.UseHangfireServer(serverOptions);

        //Ensure our recurring tasks are running if required.
        if (iniSettings.Behaviour.DoPlatformManagerInit) TpHangfireJobsManager.Initialise();
    }
}

As part of the application starting up I add a fire and forget job as follows:

BackgroundJob.Enqueue<TpHardLinkAttachmentsForEmailMessagesWatchDog>(aX => aX.ExecuteJob(null, new TpHardLinkAttachmentsForEmailMessagesWatchDogParamsInput
                                                {
                                                    ForYear = 2017
                                                }));

and the method being executed is:

public override void ExecuteJob(PerformContext aContext, TpHardLinkAttachmentsForEmailMessagesWatchDogParamsInput aParams)
{
    Trace.WriteLine($"Params: {aParams.ForYear}");
    DateTime startDate = new DateTime(aParams.ForYear, 1, 1);
    DateTime endDate = new DateTime(aParams.ForYear, 1, 31);

    //Start with a huge record id so we can work backwards
    long currRecId = int.MaxValue;

    //Keep getting records until we are done
    while (true)
    {


        //Find all records that have to be processed. We start with the highest record id in the year and then work backwards until we run out of record
        long idLast = currRecId;
        Trace.WriteLine($"Id Last: {idLast}");
        List<EmailActivityTableModel> recordsToProcess = _EmailActivityDomainService.GetWhere(
                                                                                        aR => aR.Id < idLast &&
                                                                                              aR.Created >= startDate &&
                                                                                              aR.Created <= endDate &&
                                                                                              (
                                                                                                  aR.ExternalIdKind == ExternalIdKindEnum.Family ||
                                                                                                  aR.ExternalIdKind == ExternalIdKindEnum.Individual
                                                                                              ) &&
                                                                                              (
                                                                                                  aR.ExternalId != "" ||
                                                                                                  aR.ExternalId != "-1"
                                                                                              ) //Ensure we have an external identifier
                                                                                    ).OrderByDescending(aR => aR.Id)
                                                                                    .Take(5).ToList();

        //We are done if we did not find any records
        if (recordsToProcess.Count == 0)
            break;

        aContext.WriteLine("Process {0} records.", recordsToProcess.Count);
        Trace.WriteLine($"Record Count: {recordsToProcess.Count}");

        foreach (EmailActivityTableModel emailActivityRecord in recordsToProcess)
        {
            aContext.WriteLine(emailActivityRecord.Id);

            currRecId = emailActivityRecord.Id;
            Trace.WriteLine($"Curr Rec Id: {emailActivityRecord.Id}");
        }

        //TESTING - Bail after the first iteration so 5 records
        return;
    }
}

All the method is doing is ultimately reading some records from the database and ouputting the primary key. What I am seeing is that method get called multiple times (the amount varies from each run of the console application) and I see the following:

I only see a single succeeded task in HangFire though:

I am getting no exceptions (note all 5 iterations complete each time) so the job is not getting restarted because of that.

So my question is why is my ExecuteJob() getting called so many times?

Answer 1

I am answering this for completeness.

It turned out I was getting an exception bit it was not showing in my logs. I eventually tracked it down by adding the coloured console logger.

I am using a third party package called HangFire.Console which has an issue with the attribute DisableConcurrentExecution as detailed here. Swapping that attribute for the Mutex and now all is well.