The Hangfire does not find, or resolve, the project assembly -all of it- eventhough it gets a heartbeat.
Not sure what is causing this, but the dashboard also shows that the app restarts every three minute which is impossible because we use a loadbalancer on an app that is being used by hundreds of people at the same time. This many cold-starts would mean extremely long loading times which the users would complain about.
This is the error that is shown on the recurring jobs tab;
System.InvalidOperationException: Recurring job can't be scheduled, see inner exception for details.
---> Hangfire.Common.JobLoadException: Could not load the job. See inner exception for the details.
---> System.IO.FileNotFoundException: Could not resolve assembly 'My.Api'.
at System.TypeNameParser.ResolveAssembly(String asmName, Func`2 assemblyResolver, Boolean throwOnError, StackCrawlMark& stackMark)
at System.TypeNameParser.ConstructType(Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.TypeNameParser.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError, Boolean ignoreCase, StackCrawlMark& stackMark)
at System.Type.GetType(String typeName, Func`2 assemblyResolver, Func`4 typeResolver, Boolean throwOnError)
at Hangfire.Common.TypeHelper.DefaultTypeResolver(String typeName)
at Hangfire.Storage.InvocationData.DeserializeJob()
--- End of inner exception stack trace ---
at Hangfire.Storage.InvocationData.DeserializeJob()
at Hangfire.RecurringJobEntity..ctor(String recurringJobId, IDictionary`2 recurringJob, ITimeZoneResolver timeZoneResolver, DateTime now)
--- End of inner exception stack trace ---
at Hangfire.Server.RecurringJobScheduler.ScheduleRecurringJob(BackgroundProcessContext context, IStorageConnection connection, String recurringJobId, RecurringJobEntity recurringJob, DateTime now)
Turns out, Hangfire itself does not work great when multiple apps use the same sql schema. To solve this problem I used Hangfire.MAMQSqlExtension. It is a third-party extension but the repo says that it is officially recognized by Hangfire.
If you are using the same schema for multiple apps, you have to use this extension in all your apps.
If your apps have different versions alive at the same time (e.g. production, test, development) this app itself does not fully work for failed jobs. If a job fails, regular Hangfire will not respect it's original queue, hence will move it to the default queue. Which will eventually create problems if you app only works with your app's queue or if the default queue is shared. To solve that issue, to force Hangfire to respect the original queue attribute, I used this solution. Which works great, and you get to name your app's queue depending on your web.config or appsettings.json.