I am working on a .NET Core 3.1 background service to be installed as a daemon on an Debian AWS EC2 instance.
It is important to gracefully shut down the daemon to stop running tasks and finalize a number of tasks to be handled (sending some events, etc).
The basic implementation looks like this:
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
namespace MyApp.WorkerService
{
public class Program
{
public static void Main(string[] args)
{
CreateHostBuilder(args).UseSystemd().Build().Run();
}
public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureServices((hostContext, services) =>
{
services.AddHostedService<Worker>();
});
}
}
You can see I am using the SystemdLifetime
here.
The worker is as follows:
using System;
using System.Threading;
using System.Threading.Tasks;
using AdClassifier.App.Audit;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using NLog;
namespace MyApp.WorkerService
{
public class Worker : BackgroundService
{
private static readonly ILogger Logger = LogManager.GetLogger(typeof(Worker).FullName);
private readonly int _jobPollIntervalMilliseconds;
public IServiceProvider Services { get; }
public Worker(IServiceProvider services, IConfiguration configuration)
{
Services = services;
_jobPollIntervalMilliseconds = configuration.GetValue<int>("JobPollIntervalMilliseconds");
}
protected override Task ExecuteAsync(CancellationToken stoppingToken)
{
Logger.Info("Worker running.");
var task = new Task(o => DoWork(stoppingToken), stoppingToken);
task.Start();
return task;
}
public override async Task StopAsync(CancellationToken cancellationToken)
{
Logger.Info("Worker stopping");
await base.StopAsync(cancellationToken);
Logger.Info("Worker stopped");
}
private void DoWork(CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
using (var scope = Services.CreateScope())
{
// do some work
}
Thread.Sleep(_jobPollIntervalMilliseconds);
}
Logger.Info("cancellation requested!");
}
}
}
The problem
As I mentioned, we are setting this up as a daemon, like this
[Unit]
Description=my worker
Requires=deploy-my-worker.service
After=multi-user.target deploy-my-worker.service
ConditionFileIsExecutable=/home/my-worker/current/myworker
[Service]
Environment=DOTNET_PRINT_TELEMETRY_MESSAGE=false
Environment=DOTNET_CLI_TELEMETRY_OPTOUT=true
Environment=ASPNETCORE_URLS=http://*:5000
Environment=DOTNET_ENVIRONMENT=Staging
Environment=ASPNETCORE_ENVIRONMENT=Staging
WorkingDirectory=/home/my-worker/current
ExecStart=/home/my-worker/current/myworker
SyslogIdentifier=my-worker
Restart=always
RestartSec=10
KillSignal=SIGTERM
User=usr
Group=usrgroup
[Install]
WantedBy=multi-user.target
The problem is that the worker will not stop gracefully. I am checking logs for the following log entries, but they do not appear:
Worker stopping
, cancellation requested!
, Worker stopped
Note that the application does shut down. What we have tried in order to shut down the service are the following:
systemctl stop my-worker.service
kill
kill -SIGTERM
kill -SIGINT
What Works
If I start the worker like this: usr@ip-10-168-19-126:~/my-worker/current$ ./myworker
and then press Ctrl-C
(which should be a SIGINT
), the application stops, and in my logs I can see the correct messages:
2020-05-21 16:16:57.9676|INFO|MyApp.WorkerService.Worker|Worker stopping
2020-05-21 16:16:57.9937|INFO|MyApp.WorkerService.Worker|cancellation requested!
2020-05-21 16:16:57.9937|INFO|MyApp.WorkerService.Worker|Worker stopped
2020-05-21 16:16:57.9980 Info AppDomain Shutting down. Logger closing...
Any ideas how I can get the daemon to work as expected?
NOTE:
I have good reason to believe that the problem lies somewhere in the daemon setup, or UseSystemd()
.
I replaced UseSystemd()
with UseWindowsService()
and installed it as a Windows service on a windows machine. Then went forward with starting and stopping the service via the Services panel, and saw shutdown logging as expected.
So, I am tempted to assume that there is no problem in the implementation, but rather somewhere in the setup.
It seems that the problem lay with NLog's shutdown. This was fixed by doing the following: LogManager.AutoShutdown = false;
and in Worker::StopAsync
adding LogManager.Shutdown();