Threads increase abnormally in linux service
I have a service that runs in linux under SystemD but gets compiled and debugged in VS22 under Windows. The service is mainly a proxy to a MariaDB10 database shaped as a BackgroundWorker serving clients via SignalR. If I run it in relase mode on Windows, the number of logical threads remains in a reasonable value (20-25 approx). See pic below.
Under linux, after few minutes (i cannot give you more insight unfortuantely... i still have to figure out what could be changing) the number of threads start increasing constantly every second.
see pic here arriving already to more than 100 and still counting:
Reading current logical threads increasing / thread stack is leaking i got confirmed that the CLR is allowing new threads if the others are not completing, but there is currently no change in the code when moving from Windows to Linux.
This is the HostBuilder with the call to SystemD
public static IHostBuilder CreateWebHostBuilder(string[] args)
{
string curDir = MondayConfiguration.DefineCurrentDir();
IConfigurationRoot config = new ConfigurationBuilder()
// .SetBasePath(Directory.GetCurrentDirectory())
.SetBasePath(curDir)
.AddJsonFile("servicelocationoptions.json", optional: false, reloadOnChange: true)
#if DEBUG
.AddJsonFile("appSettings.Debug.json")
#else
.AddJsonFile("appSettings.json")
#endif
.Build();
return Host.CreateDefaultBuilder(args)
.UseContentRoot(curDir)
.ConfigureAppConfiguration((_, configuration) =>
{
configuration
.AddIniFile("appSettings.ini", optional: true, reloadOnChange: true)
#if DEBUG
.AddJsonFile("appSettings.Debug.json")
#else
.AddJsonFile("appSettings.json")
#endif
.AddJsonFile("servicelocationoptions.json", optional: false, reloadOnChange: true);
})
.UseSerilog((_, services, configuration) => configuration
.ReadFrom.Configuration(config, sectionName: "AppLog")// (context.Configuration)
.ReadFrom.Services(services)
.Enrich.FromLogContext()
.WriteTo.Console())
// .UseSerilog(MondayConfiguration.Logger)
.ConfigureServices((hostContext, services) =>
{
services
.Configure<ServiceLocationOptions>(hostContext.Configuration.GetSection(key: nameof(ServiceLocationOptions)))
.Configure<HostOptions>(opts => opts.ShutdownTimeout = TimeSpan.FromSeconds(30));
})
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
ServiceLocationOptions locationOptions = config.GetSection(nameof(ServiceLocationOptions)).Get<ServiceLocationOptions>();
string url = locationOptions.HttpBase + "*:" + locationOptions.Port;
webBuilder.UseUrls(url);
})
.UseSystemd();
}
In the meantime I am trying to trace all the Monitor.Enter() that I use to render serial the API endpoints that touch the state of the service and the inner structures, but in Windows seems all ok.
I am starting wondering if the issue in the call to SystemD. I would like to know what is really involved in a call to UseSystemD() but there is not so much documentation around.
I did just find [https://devblogs.microsoft.com/dotnet/net-core-and-systemd/] (https://devblogs.microsoft.com/dotnet/net-core-and-systemd/) by Glenn Condron and few quick notes on MSDN.
EDIT 1: To debug further I created a class to scan the threadpool using ClrMd. My main service has an heartbeat (weird it is called Ping) as follows (not the add to processTracker.Scan()):
private async Task Ping()
{
await _containerServer.SyslogQueue.Writer.WriteAsync((
LogLevel.Information,
$"Monday Service active at: {DateTime.UtcNow.ToLocalTime()}"));
string processMessage = ProcessTracker.Scan();
await _containerServer.SyslogQueue.Writer.WriteAsync((LogLevel.Information, processMessage));
_logger.DebugInfo()
.Information("Monday Service active at: {Now}", DateTime.UtcNow.ToLocalTime());
}
where the processTrackes id constructed like this:
public static class ProcessTracker
{
static ProcessTracker()
{
}
public static string Scan()
{
// see https://stackoverflow.com/questions/31633541/clrmd-throws-exception-when-creating-runtime/31745689#31745689
StringBuilder sb = new();
string answer = $"Active Threads{Environment.NewLine}";
// Create the data target. This tells us the versions of CLR loaded in the target process.
int countThread = 0;
var pid = Process.GetCurrentProcess().Id;
using (var dataTarget = DataTarget.AttachToProcess(pid, 5000, AttachFlag.Passive))
{
// Note I just take the first version of CLR in the process. You can loop over
// every loaded CLR to handle the SxS case where both desktop CLR and .Net Core
// are loaded in the process.
ClrInfo version = dataTarget.ClrVersions[0];
var runtime = version.CreateRuntime();
// Walk each thread in the process.
foreach (ClrThread thread in runtime.Threads)
{
try
{
sb = new();
// The ClrRuntime.Threads will also report threads which have recently
// died, but their underlying data structures have not yet been cleaned
// up. This can potentially be useful in debugging (!threads displays
// this information with XXX displayed for their OS thread id). You
// cannot walk the stack of these threads though, so we skip them here.
if (!thread.IsAlive)
continue;
sb.Append($"Thread {thread.OSThreadId:X}:");
countThread++;
// Each thread tracks a "last thrown exception". This is the exception
// object which !threads prints. If that exception object is present, we
// will display some basic exception data here. Note that you can get
// the stack trace of the exception with ClrHeapException.StackTrace (we
// don't do that here).
ClrException? currException = thread.CurrentException;
if (currException is ClrException ex)
sb.AppendLine($"Exception: {ex.Address:X} ({ex.Type.Name}), HRESULT={ex.HResult:X}");
// Walk the stack of the thread and print output similar to !ClrStack.
sb.AppendLine(" ------> Managed Call stack:");
var collection = thread.EnumerateStackTrace().ToList();
foreach (ClrStackFrame frame in collection)
{
// Note that CLRStackFrame currently only has three pieces of data:
// stack pointer, instruction pointer, and frame name (which comes
// from ToString). Future versions of this API will allow you to get
// the type/function/module of the method (instead of just the
// name). This is not yet implemented.
sb.AppendLine($" {frame}");
}
}
catch
{
//skip to the next
}
finally
{
answer += sb.ToString();
}
}
}
answer += $"{Environment.NewLine} Total thread listed: {countThread}";
return answer;
}
}
All fine, in Windows it prints a lot of nice information in some kind of tree textual view.
The point is that somewhere it requires Kernel32.dll and in linux that is not available. Can someone give hints on this? The service is published natively without .NET infrastructure, in release mode, arch linux64, single file.
thanks a lot Alex
I found a way to skip the whole logging of what I needed from a simple debug session. I was not aware I could attach also to a Systemd process remotely. Just followed https://learn.microsoft.com/en-us/visualstudio/debugger/remote-debugging-dotnet-core-linux-with-ssh?view=vs-2022 for a quick step by step guide. The only preresquisites are to let the service be in debug mode and have the NET runtime installed on the host, but that's really all. Sorry for not having known this earlier.
Alex