Hangfire ContinueWithJob is stuck in awaiting state, though parent job has succeeded

Question

I have a few jobs executed one after the other via ContinueJobWith<MyHandler>(parentJobId, x => x.DoWork()).

However, the second job is not getting processed and always sits in Awaiting state:

The job itself is like this:

Why this can happen and where to check for resultion?

• We are using Autofac as DI container, but we have our own JobActivator implementation because we have to deal with multitenancy.
• We are using SQL Server 2019 for storage.
• Hangfire version is 1.7.10
• This is MVC 5 application
• I've not seen any errors/exceptions in any logs or during debugging
• After going through this I've added this to our Autofac registration

            builder.RegisterType<BackgroundJobStateChanger>()
                   .As<IBackgroundJobStateChanger>()
                   .InstancePerLifetimeScope();

This made no difference.

This is how the jobs are executed:

var parentJobId = _backgroundJobClient.Schedule<Handler>(h => h.ConvertCertToTraining(certId, command.SetUpOneToOneRelationship), TimeSpan.FromSeconds(1));
var filesCopyJObId = _backgroundJobClient.ContinueJobWith<Handler>(parentJobId, h => h.CopyAttachedFiles());
_backgroundJobClient.ContinueJobWith<Handler>(filesCopyJObId, h => h.NotifyUser(command.CertificationToBeConvertedIds, _principal.GetEmail()));

All the parameters are either int, bool or string. If I enqueue the awaiting jobs by hand, they are executed without issues.

I've added Hangfire logging, but could not see any issues there: server starts, stops, jobs change status, but could not see any obvious errors there.

What other things I should consider or where/how should I debug this?

Answer 1

During the investigation, it turned out that we were replacing JobFilterProviderCollection with our own collection:

    var filterProviderCollection = new JobFilterProviderCollection
    {
        new MyFilterProvider(...)
    };
    var backgroundJobClient = new BackgroundJobClient(JobStorage.Current, filterProviderCollection);

MyFilterProvider looked like this:

    public IEnumerable<JobFilter> GetFilters(Job job)
    {
        return new JobFilter[]
        {
             new JobFilter(new HangfireTenantFilter(_tenantDetail, _principal), JobFilterScope.Global,  null),
             new JobFilter(new HangfireFunctionalityFilter(_functionalityFilter), JobFilterScope.Global, null),
        };
    }

It turned out that code that was doing work on Continuation only took filters from this filter collection and ContinuationsSupportAttribute was not executed there in the right time. So re-adding default Hangfire filters from GlobalJobFilters.Filters fixed the situation:

    public IEnumerable<JobFilter> GetFilters(Job job)
    {
        var customFilters = new List<JobFilter>()
        {
             new JobFilter(new HangfireTenantFilter(_tenantDetail, _principal), JobFilterScope.Global,  null),
             new JobFilter(new HangfireFunctionalityFilter(_functionalityFilter), JobFilterScope.Global, null),
        };
        customFilters.AddRange(GlobalJobFilters.Filters);

        return customFilters;
    }