I'm the developer in charge of a C# web application running on IIS 10.0 and I use the FluentScheduler library to schedule my jobs.
This job does a database query and then generates some files. Recently our jobs have been failing as they were running for too long and Windows kills the thread (this only happens on specific days with a large influx of data to be processed).
After doing some optimizations on the database access I got the killing of the thread down a bunch but it still occasionally happens.
The problem is that after having it's thread killed (and logging the exception), the job stops running on it's scheduled time.
How can I make sure the job keeps running even if this exception does happen?
My code below:
Schedule(new GenerateFiles()).NonReentrant().ToRunOnceAt(DateTime.Now.AddMinutes(10)).AndEvery(30).Seconds();
The 10-minute delay on the first run is there because we cache some information available on the database to improve the application's performance and this job uses data from that cache and I don't know how to make it only begin after the cache is done so I added this delay.
Any other exceptions caught and logged on my jobs does not cause this issue. It's only when the thread runs for too long and Windows kills it that it stops running again.
Edit: Adding the line in which the application fails (at least that's what the Stack Trace tells me.
The entire job is quite extensive and I can't really post it here.
foreach (var datumToGenerate in context.GenerateData.Include(f => f.Datum))
{
var datum = datumToGenerate.Datum;
if (!datum.Generated)
{
output.Add(datum);
i++;
if (i == 100) return output;
}
}
As you can see, I lowered the number of entries to be processed at a time to 100 but even setting it as 50 or a low number, I get the error eventually as the GenerateData table is quite large even though its entries get deleted after being processed.
Edit2: The code, in fact, fails on any random part of the class. It runs fine for around 10 minutes and then it just crashes. Am I simply screwed??
Me and my colleague found the answer to the issue.
It had nothing to do with code.
IIS's application pool has a default sleep timeout of 20 minutes. We disabled the application pool's timeout by setting its value to 0 and never again did that exception occur.