glibc Arena system_mem does not add up to the amount of VSS or RSS

Question

I have a process where I'm trying to account for where all the memory is going. My first thought was to iterate through all the arenas with gdb and output their system memory like so (I've configured the process to 32 arenas):

define arenaSizeWalk
  set var $n = main_arena
  set var $size = 0
  set var $count = 0
  while $count < 32
    set var $size = $n.system_mem
    set var $n = $n.next
    set var $count = $count + 1
    printf "%d\n", $size
  end
end

However, this comes up quite short. This adds up to ~3GB. But my total memory footprint as gleaned from pmap (excluding stack guards and loaded libraries) is to the tune of 25GB.

Moreover, I used gdb-heap to dump all the memory chunks and compared them to the output from pmap

From heap all:
...
39: 0x00007fc1840c8000 -> 0x00007fc1841c8fff  inuse: 1052672 bytes (<MChunkPtr chunk=0x7fc1840c8000 mem=0x7fc1840c8010 prev_size=0 IS_MMAPPED chunksize=1052672 memsize=1052656>)
40: 0x00007fc1841c9000 -> 0x00007fc184649fff  inuse: 4722688 bytes (<MChunkPtr chunk=0x7fc1841c9000 mem=0x7fc1841c9010 prev_size=0 IS_MMAPPED chunksize=4722688 memsize=4722672>)
41: 0x00007fc18464a000 -> 0x00007fc18474afff  inuse: 1052672 bytes (<MChunkPtr chunk=0x7fc18464a000 mem=0x7fc18464a010 prev_size=0 IS_MMAPPED chunksize=1052672 memsize=1052656>)
42: 0x00007fc1d401b000 -> 0x00007fc1d4044fff  inuse: 172032 bytes (<MChunkPtr chunk=0x7fc1d401b000 mem=0x7fc1d401b010 prev_size=0 IS_MMAPPED chunksize=172032 memsize=172016>)
...
From pmap:
00007fc1840c8000   6668K rw---   [ anon ]
00007fc18474c000   1024K rw---   [ anon ]
00007fc18484d000   1024K rw---   [ anon ]
00007fc18494e000   1024K rw---   [ anon ]
00007fc184a4f000   1024K rw---   [ anon ]
00007fc184b50000   1024K rw---   [ anon ]
00007fc184c51000   1024K rw---   [ anon ]
00007fc184d52000   1024K rw---   [ anon ]
00007fc184e53000   1024K rw---   [ anon ]
00007fc184f54000   1024K rw---   [ anon ]

Chunk 42 there leaped in address space from 7fc18... -> 7fc18d. Meanwhile, we have these pmapped addresses proceeding after 07fc1840c8000 that don't seem to be accounted for with existing chunks.

I'm using glibc 2.17 under RHEL7. Am I missing something??? How can I have memory regions that don't seem to be accounted for by the system default malloc implementation? Or perhaps I've grossly misunderstood something somewheres. And how am I so short on the memory accounting?

Thanks in advance!!

Answer 1

I figured this out. The reason why memory regions were not traceable in this manner was because some other parts of the program were calling mmap() directly as opposed to using malloc() or new.

Only those allocations that interface with malloc will be traceable by examining the main_arena or utilizing gdb-heap.