I am trying to understand how !address -summary report managed allocations. I have following code to try to test it out. I basically change the size of array and then attach windbg to inspect results for !address -summary
static void Main(string[] args)
{
int size = 2000;
var something = new byte[size];
for (int i = 0; i < something.Length; i++)
{
something[i] = 10;
}
Console.WriteLine("Attach Windbg now");
Console.Read();
// not sure if its needed but have it so that this object is still referenced and optimizations don't remove it proactively
for (int i = 0; i < something.Length; i++)
{
Console.WriteLine(something[i]);
}
}
With a size = 50000
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 53 7fff`e1546000 ( 128.000 Tb) 100.00%
<unknown> 61 0`18a1d000 ( 394.113 Mb) 80.32% 0.00%
Image 163 0`031da000 ( 49.852 Mb) 10.16% 0.00%
MappedFile 12 0`019f8000 ( 25.969 Mb) 5.29% 0.00%
Stack 12 0`01000000 ( 16.000 Mb) 3.26% 0.00%
Heap 22 0`002f3000 ( 2.949 Mb) 0.60% 0.00%
Other 8 0`001bf000 ( 1.746 Mb) 0.36% 0.00%
TEB 4 0`00008000 ( 32.000 kb) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 101 0`19d0b000 ( 413.043 Mb) 84.18% 0.00%
MEM_IMAGE 163 0`031da000 ( 49.852 Mb) 10.16% 0.00%
MEM_MAPPED 19 0`01bc5000 ( 27.770 Mb) 5.66% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 53 7fff`e1546000 ( 128.000 Tb) 100.00%
MEM_RESERVE 51 0`1aec5000 ( 430.770 Mb) 87.79% 0.00%
MEM_COMMIT 232 0`03be5000 ( 59.895 Mb) 12.21% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ 24 0`0262d000 ( 38.176 Mb) 7.78% 0.00%
PAGE_READONLY 71 0`00dfb000 ( 13.980 Mb) 2.85% 0.00%
PAGE_WRITECOPY 31 0`00417000 ( 4.090 Mb) 0.83% 0.00%
PAGE_READWRITE 81 0`00335000 ( 3.207 Mb) 0.65% 0.00%
PAGE_EXECUTE_WRITECOPY 3 0`00043000 ( 268.000 kb) 0.05% 0.00%
PAGE_EXECUTE_READWRITE 15 0`00016000 ( 88.000 kb) 0.02% 0.00%
PAGE_READWRITE|PAGE_GUARD 5 0`00015000 ( 84.000 kb) 0.02% 0.00%
PAGE_EXECUTE 2 0`00003000 ( 12.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0`7fff0000 7ff5`7f760000 ( 127.959 Tb)
<unknown> 0`02e62000 0`0ffde000 ( 255.867 Mb)
Image 7ffd`0ab5d000 0`010e4000 ( 16.891 Mb)
MappedFile 0`0150c000 0`01234000 ( 18.203 Mb)
Stack 0`1ba90000 0`003f9000 ( 3.973 Mb)
Heap 0`02741000 0`00101000 ( 1.004 Mb)
Other 0`011b0000 0`00181000 ( 1.504 Mb)
TEB 7ff5`ff928000 0`00002000 ( 8.000 kb)
PEB 7ff5`ff926000 0`00001000 ( 4.000 kb)
this is how it looks in !dumpheap -stat output
00007ffd0ad95740 4 50609 System.Byte[]
this is how it looks in !EEHeap -gc output
Number of GC Heaps: 1
generation 0 starts at 0x0000000002ec1030
generation 1 starts at 0x0000000002ec1018
generation 2 starts at 0x0000000002ec1000
ephemeral segment allocation context: ephemeral segment allocation context: none
none
segment begin allocated size
0000000002ec0000 0000000002ec1000 0000000002ed3fe8 0x12fe8(77800)
Large object heap starts at 0x0000000012ec1000
segment begin allocated size
0000000012ec0000 0000000012ec1000 0000000012ec9720 0x8720(34592)
Total Size: Total Size: Size: 0x1b708 (112392) bytesSize: 0x1b708 (112392) bytes.
.
------------------------------
------------------------------
GC Heap Size: GC Heap Size: Size: 0x1b708 (112392) bytesSize: 0x1b708 (112392) bytes.
With a size = 2000
0:003> !address -summary
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 50 7fff`e1941000 ( 128.000 Tb) 100.00%
<unknown> 57 0`1861b000 ( 390.105 Mb) 80.16% 0.00%
Image 163 0`031da000 ( 49.852 Mb) 10.24% 0.00%
MappedFile 12 0`019f8000 ( 25.969 Mb) 5.34% 0.00%
Stack 12 0`01000000 ( 16.000 Mb) 3.29% 0.00%
Heap 22 0`002fa000 ( 2.977 Mb) 0.61% 0.00%
Other 8 0`001bf000 ( 1.746 Mb) 0.36% 0.00%
TEB 4 0`00008000 ( 32.000 kb) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 97 0`19910000 ( 409.063 Mb) 84.05% 0.00%
MEM_IMAGE 163 0`031da000 ( 49.852 Mb) 10.24% 0.00%
MEM_MAPPED 19 0`01bc5000 ( 27.770 Mb) 5.71% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 50 7fff`e1941000 ( 128.000 Tb) 100.00%
MEM_RESERVE 50 0`1aadc000 ( 426.859 Mb) 87.71% 0.00%
MEM_COMMIT 229 0`03bd3000 ( 59.824 Mb) 12.29% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ 24 0`0262d000 ( 38.176 Mb) 7.84% 0.00%
PAGE_READONLY 71 0`00dfb000 ( 13.980 Mb) 2.87% 0.00%
PAGE_WRITECOPY 31 0`00417000 ( 4.090 Mb) 0.84% 0.00%
PAGE_READWRITE 79 0`00326000 ( 3.148 Mb) 0.65% 0.00%
PAGE_EXECUTE_WRITECOPY 3 0`00043000 ( 268.000 kb) 0.05% 0.00%
PAGE_EXECUTE_READWRITE 15 0`00016000 ( 88.000 kb) 0.02% 0.00%
PAGE_READWRITE|PAGE_GUARD 4 0`00012000 ( 72.000 kb) 0.01% 0.00%
PAGE_EXECUTE 2 0`00003000 ( 12.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0`7fff0000 7ff5`7f7a0000 ( 127.959 Tb)
<unknown> 0`02912000 0`0ffee000 ( 255.930 Mb)
Image 7ffd`0ab5d000 0`010e4000 ( 16.891 Mb)
MappedFile 0`0102c000 0`01234000 ( 18.203 Mb)
Stack 0`1b350000 0`003f9000 ( 3.973 Mb)
Heap 0`022a8000 0`00101000 ( 1.004 Mb)
Other 0`00cd0000 0`00181000 ( 1.504 Mb)
TEB 7ff5`ff968000 0`00002000 ( 8.000 kb)
PEB 7ff5`ff967000 0`00001000 ( 4.000 kb)
this is how it looks in !dumpheap -stat output
00007ffd0ad95740 4 2609 System.Byte[]
here is the ouput of !EEHeap -gc
Number of GC Heaps: 1
generation 0 starts at 0x0000000002bd1030
generation 1 starts at 0x0000000002bd1018
generation 2 starts at 0x0000000002bd1000
ephemeral segment allocation context: none
segment begin allocated size
0000000002bd0000 0000000002bd1000 0000000002bd7fe8 0x6fe8(28648)
Large object heap starts at 0x0000000012bd1000
segment begin allocated size
0000000012bd0000 0000000012bd1000 0000000012bd9720 0x8720(34592)
Total Size: Size: 0xf708 (63240) bytes.
------------------------------
GC Heap Size: Size: 0xf708 (63240) bytes.
When I change the array size to 5000000 this is what I see in !dumpheap -stat
00007ffd0ad95740 4 5000609 System.Byte[]
Here is output of !EEHeap -gc command
Number of GC Heaps: 1
generation 0 starts at 0x0000000002bd1030
generation 1 starts at 0x0000000002bd1018
generation 2 starts at 0x0000000002bd1000
ephemeral segment allocation context: none
segment begin allocated size
0000000002bd0000 0000000002bd1000 0000000002bd7fe8 0x6fe8(28648)
Large object heap starts at 0x0000000012bd1000
segment begin allocated size
0000000012bd0000 0000000012bd1000 000000001309e298 0x4cd298(5034648)
Total Size: Size: 0x4d4280 (5063296) bytes.
------------------------------
GC Heap Size: Size: 0x4d4280 (5063296) bytes.
And address summary is below
0:003> !address -summary
Mapping file section regions...
Mapping module regions...
Mapping PEB regions...
Mapping TEB and stack regions...
Mapping heap regions...
Mapping page heap regions...
Mapping other regions...
Mapping stack trace database regions...
Mapping activation context regions...
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
Free 50 7fff`e153a000 ( 128.000 Tb) 100.00%
<unknown> 61 0`18a1d000 ( 394.113 Mb) 80.31% 0.00%
Image 163 0`031da000 ( 49.852 Mb) 10.16% 0.00%
MappedFile 12 0`019f8000 ( 25.969 Mb) 5.29% 0.00%
Stack 12 0`01000000 ( 16.000 Mb) 3.26% 0.00%
Heap 22 0`002ff000 ( 2.996 Mb) 0.61% 0.00%
Other 8 0`001bf000 ( 1.746 Mb) 0.36% 0.00%
TEB 4 0`00008000 ( 32.000 kb) 0.01% 0.00%
PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
--- Type Summary (for busy) ------ RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_PRIVATE 101 0`19d17000 ( 413.090 Mb) 84.18% 0.00%
MEM_IMAGE 163 0`031da000 ( 49.852 Mb) 10.16% 0.00%
MEM_MAPPED 19 0`01bc5000 ( 27.770 Mb) 5.66% 0.00%
--- State Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal
MEM_FREE 50 7fff`e153a000 ( 128.000 Tb) 100.00%
MEM_RESERVE 51 0`1aa05000 ( 426.020 Mb) 86.82% 0.00%
MEM_COMMIT 232 0`040b1000 ( 64.691 Mb) 13.18% 0.00%
--- Protect Summary (for commit) - RgnCount ----------- Total Size -------- %ofBusy %ofTotal
PAGE_EXECUTE_READ 24 0`0262d000 ( 38.176 Mb) 7.78% 0.00%
PAGE_READONLY 71 0`00e1d000 ( 14.113 Mb) 2.88% 0.00%
PAGE_READWRITE 81 0`007df000 ( 7.871 Mb) 1.60% 0.00%
PAGE_WRITECOPY 31 0`00417000 ( 4.090 Mb) 0.83% 0.00%
PAGE_EXECUTE_WRITECOPY 3 0`00043000 ( 268.000 kb) 0.05% 0.00%
PAGE_EXECUTE_READWRITE 15 0`00016000 ( 88.000 kb) 0.02% 0.00%
PAGE_READWRITE|PAGE_GUARD 5 0`00015000 ( 84.000 kb) 0.02% 0.00%
PAGE_EXECUTE 2 0`00003000 ( 12.000 kb) 0.00% 0.00%
--- Largest Region by Usage ----------- Base Address -------- Region Size ----------
Free 0`7fff0000 7ff5`7f7d0000 ( 127.959 Tb)
<unknown> 0`02be2000 0`0ffee000 ( 255.930 Mb)
Image 7ffd`0ab5d000 0`010e4000 ( 16.891 Mb)
MappedFile 0`0134c000 0`01214000 ( 18.078 Mb)
Stack 0`1b760000 0`003f9000 ( 3.973 Mb)
Heap 0`0265d000 0`00101000 ( 1.004 Mb)
Other 0`00fd0000 0`00181000 ( 1.504 Mb)
TEB 7ff5`ff996000 0`00002000 ( 8.000 kb)
PEB 7ff5`ff99e000 0`00001000 ( 4.000 kb)
The output of dumpheap and eeheap totally makes sense. I don't understand where these managed allocation should be displayed in !address -summary
Please try the following code, which allocates 20 MB:
static void Main(string[] args)
{
Console.WriteLine("Attach Windbg now");
Console.ReadLine();
int size = 20000000;
var something = new byte[size];
Console.WriteLine("Look again");
Console.ReadLine();
// not sure if its needed but have it so that this object is still referenced and optimizations don't remove it proactively
something[0] = 1;
}
Right at the beginning my machine shows
<unknown> 49 0`19502000 ( 405.008 Mb) 82.39% 0.00%
MEM_RESERVE 45 0`1b0d4000 ( 432.828 Mb) 88.05% 0.01%
MEM_COMMIT 240 0`03ac0000 ( 58.750 Mb) 11.95% 0.00%
in the output of !address -summary.
Now do a
.loadby sos clr
bp kernel32!VirtualAlloc
g
and press Enter in the console application. It will hit the breakpoint and you will see
0:000> k
Child-SP RetAddr Call Site
00000000`001ce2b8 000007fe`f77c265a KERNEL32!VirtualAlloc
...
00000000`001ce5f0 000007fe`f77c7ece clr!WKS::GCHeap::Alloc+0x348
00000000`001ce640 000007fe`981600f7 clr!JIT_NewArr1+0x49e
...
0:000> !clrstack
OS Thread Id: 0x5c4 (0)
Child SP IP Call Site
00000000001ce778 0000000076cb5c68 [HelperMethodFrame: 00000000001ce778]
00000000001ce8f0 000007fe981600f7 ManagedVirtualAlloc.Program.Main(System.String[])
00000000001cec30 000007fef7829e03 [GCFrame: 00000000001cec30]
So you can see that from the Main method the .NET runtime tries to allocate virtual memory to fit the new byte array (as indicated by JIT_NewArr1).
However, looking at the parameters passed to VirtualAlloc() (note that kb is misleading for 64 bit), we can see
0:000> r
...
rcx=0000000012392000 // address of an already reserved memory region
rdx=000000000130b000 // size: 19968000
r8=0000000000001000 // MEM_COMMIT
r9=0000000000000004 // PAGE_READWRITE
...
0:000> kb; *** Warning: misleading arguments for 64 bit!
RetAddr : Args to Child : Call Site
000007fe`f77c265a : 00000000`00000000 00000000`00000000 00000000`001ce369 000007fe`00000000 : KERNEL32!VirtualAlloc
we see that it specifies an address which was reserved before and shall now be committed. After the call (enter g), we see that some (~ 20 MB) reserved memory has been converted to committed memory while the <unknown> amount is still the same.
<unknown> 49 0`19502000 ( 405.008 Mb) 82.39% 0.00%
MEM_RESERVE 45 0`19dc1000 ( 413.754 Mb) 84.17% 0.00%
MEM_COMMIT 238 0`04dd3000 ( 77.824 Mb) 15.83% 0.00%
Conclusion:
.NET reserves a lot of memory in advance. That memory will already show up as <unused> right after starting the application. It will then work with that reserved and committed memory if possible, so not every single allocation results in a new VirtualAlloc() call.
Some more internals:
We also see that the .NET framework is quite clever: it allocated only 19968000 bytes instead of 20000000 bytes, because it was able reuse some bytes of a memory block which already was in committed state. Looking at the pointers:
VirtualAlloc pointer: 0x0000000012392000
"something" variable: 0x0000000012389710
The something variable points to a memory region in front of the memory block returned by the VirtualAlloc() call.