Search code examples
.netlinuxdebugging.net-corelldb

.NET Core on Linux - LLDB, SOS plugin - diagnosing memory issue


I have a .NET Core app which uses up to ~150MB on Windows, but ~1-2GB on Linux (not VM size, but actual private bytes). It calls Azure HTTP APIs in a loop to retrieve stats.

I'd like to diagnose the problem on Linux, but I don't know the tools.

I have a dump via gdump, and have been following https://codeblog.dotsandbrackets.com/net-core-memory-linux/ . In the case below, it was using just under a gig.

I can get to this point:

(lldb) sos DumpHeap -stat -min 10000
Statistics:
              MT    Count    TotalSize Class Name
00007f88ebedf438        2       131120 Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry[]
00007f88ebed5438        1       202080 System.Collections.Generic.Dictionary`2+Entry[[System.String, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry, Microsoft.EntityFrameworkCore]][]
00007f88ebec6328        1       202080 System.Collections.Generic.Dictionary`2+Entry[[System.Object, System.Private.CoreLib],[Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalEntityEntry, Microsoft.EntityFrameworkCore]][]
00007f88e8c92660        4       432604 System.Byte[]
00007f88e8cdf228        4       865120 System.String
0000000001e2d2c0       36      6008920      Free

(without the -min option, I get too much output to read/do anything with)

The ~600MB free is suspicious, but I don't know what it means, how to remedy, or any other SOS commands to try next. The SOS docs (https://github.com/dotnet/coreclr/blob/master/src/ToolBox/SOS/Strike/sosdocs.txt) look great, but my level of understanding isn't high enough to put together a diagnostic process here yet.

(Some context: I'm running on Kubernetes, have a RAM request and limit set of ~2GB, but this process keeps hitting it. As I say, on Windows it uses way less RAM. Weird.)

Edit: running without -min does thankfully end with the biggest memory users it seems. So here's the end of it:

...
00007f88ebec5de0     4727       340344 Microsoft.EntityFrameworkCore.ChangeTracking.Internal.InternalClrEntityEntry
00007f88e8c91d00     6484       347528 System.Int32[]
00007f88e9e9e0b8     9434       377360 UNKNOWN
00007f88ea4222d8     4836       386880 Microsoft.Azure.Management.Monitor.Fluent.Models.MetricDefinition
00007f88ebe47f28     9672       464256 Microsoft.Azure.Management.Monitor.Fluent.Models.MetricAvailability
00007f88e9e9f110     9433       528248 UNKNOWN
00007f88e9e9e2f8     9434       528304 UNKNOWN
00007f88ebe42a78     4836       580320 Microsoft.Azure.Management.Monitor.Fluent.Models.MetricDefinitionImpl
00007f88ea423158     3837      1074360 azure2elasticstack.Program+<ProcessMetricAsync>d__10
00007f88e8c92660     1832      2315924 System.Byte[]
00007f88e8cdf228    44713     10688048 System.String
0000000001e2d2c0    22762     22762768      Free
Total 289009 objects
(lldb)

Repo here: https://github.com/kierenj/proto-azure2elasticstack/tree/master


Solution

  • The memory usage is way down and into a respectable range for .NET Core 2.1 (netcoreapp2.1).