Hangfire - Recurring job can’t be scheduled, see inner exception for details

Question

I have an app; which is live on three different servers, using a loadbalancer for user distribution. The app uses its own queue and I have added a filter for jobs to keep their original queue in case they fail at some point. But then again, it continues to act like the app is not running. The error is like below;

System.InvalidOperationException: Recurring job can't be scheduled, see inner exception for details.
 ---> Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
 ---> System.IO.FileNotFoundException: Could not resolve assembly 'My.Api'.
   at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark)
   at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
   at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
   at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName)
   at Hangfire.Storage.InvocationData.DeserializeJob()
   --- End of inner exception stack trace ---
   at Hangfire.Storage.InvocationData.DeserializeJob()
   at Hangfire.RecurringJobEntity..ctor(String recurringJobId, IDictionary`2 recurringJob, ITimeZoneResolver timeZoneResolver, DateTime now)
   --- End of inner exception stack trace ---
   at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now)
What can be the issue here? The apps are running. And once I trigger the recurring jobs, they are good to go, until they show the above error.

This is my AppStart file;

private IEnumerable<IDisposable> GetHangfireServers()
{
    Hangfire.GlobalConfiguration.Configuration
        .SetDataCompatibilityLevel(CompatibilityLevel.Version_170)
        .UseSimpleAssemblyNameTypeSerializer()
        .UseRecommendedSerializerSettings()
        .UseSqlServerStorage(HangfireServer, new SqlServerStorageOptions
        {
            CommandBatchMaxTimeout = TimeSpan.FromMinutes(5),
            SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5),
            QueuePollInterval = TimeSpan.Zero,
            UseRecommendedIsolationLevel = true,
            DisableGlobalLocks = true
        });

    yield return new BackgroundJobServer(new BackgroundJobServerOptions {
        Queues = new[] { "myapp" + GetEnvironmentName() },
        ServerName = "MyApp" + ConfigurationHelper.GetAppSetting("Environment")
    });
}

public void Configuration(IAppBuilder app)
{
    var container = new Container();
    container.Options.DefaultScopedLifestyle = new AsyncScopedLifestyle();
    
    RegisterTaskDependencies(container);
    container.RegisterWebApiControllers(System.Web.Http.GlobalConfiguration.Configuration);
    container.Verify();
    
    var configuration = new HttpConfiguration();
    configuration.DependencyResolver = new SimpleInjectorWebApiDependencyResolver(container);
    
    /* HANGFIRE CONFIGURATION */
    if (Environment == "Production")
    {
        GlobalJobFilters.Filters.Add(new PreserveOriginalQueueAttribute());
        Hangfire.GlobalConfiguration.Configuration.UseActivator(new SimpleInjectorJobActivator(container));
        Hangfire.GlobalConfiguration.Configuration.UseLogProvider(new Api.HangfireArea.Helpers.CustomLogProvider(container.GetInstance<Core.Modules.LogModule>()));
        app.UseHangfireAspNet(GetHangfireServers);
        app.UseHangfireDashboard("/hangfire", new DashboardOptions
        {
            Authorization = new[] { new DashboardAuthorization() },
            AppPath = GetBackToSiteURL(),
            DisplayStorageConnectionString = false
        });
        AddOrUpdateJobs();
    }
    /* HANGFIRE CONFIGURATION */
    
    app.UseWebApi(configuration);
    
    WebApiConfig.Register(configuration);

}

public static void AddOrUpdateJobs()
{
    var queueName = "myapp" + GetEnvironmentName();
    RecurringJob.AddOrUpdate<HangfireArea.BackgroundJobs.AttachmentCreator>(
         "MyApp_MyTask",
         (service) => service.RunMyTask(), 
      "* * * * *", queue: queueName, timeZone: TimeZoneInfo.FindSystemTimeZoneById("Turkey Standard Time"));
}

What can be the problem here?

Answer 1

Turns out, Hangfire itself does not work great when multiple apps use the same sql schema. To solve this problem I used Hangfire.MAMQSqlExtension. It is a third-party extension but the repo says that it is officially recognized by Hangfire. If you are using the same schema for multiple apps, you have to use this extension in all your apps, otherwise you'll face the error mentioned above.

If your apps have different versions alive at the same time (e.g. production, test, development) this app itself does not fully work for failed jobs. If a job fails, regular Hangfire will not respect it's original queue, hence will move it to the default queue. Which will eventually create problems if your app only works with your app's queue or if the default queue is shared. To solve that issue, to force Hangfire to respect the original queue attribute, I used this solution. Which works great, and you get to name your app's queue depending on your web.config or appsettings.json.

My previous answer was deleted for some reason? This solves the problem and there's no other way. Do not delete the answer, for people who will experience this issue.