Search code examples
c++linuxmemory-leaksvalgrindnvidia-jetson

Running valgrind on NVIDIA Jetson gives no leak source information


tl;dr

valgrind not showing reachable memory leak source

details

C++ application was built using cmake with following extra options:

set(CMAKE_CXX_FLAGS_DEBUG "-ggdb3 -O0")
set(CMAKE_C_FLAGS_DEBUG "-ggdb3 -O0")

which were passed as seen from make VERBOSE=1 command.

Output from running /usr/bin/valgrind --num-callers=500 --trace-children=yes --leak-check=full --show-reachable=yes -v --track-origins=yes --show-leak-kinds=all ./aplication --application params indicated that relevant symbols were loaded.

To check memory leakage valgrind was used in combination with gdb which alowed checking valgrind reports after arbitrary time intervals. These reports indicated gradual reachable memory increase - indicating leakage.

The problem is that valgrind didn't provide any usable insight what might be causing memory leaks. The output is as follows:

==21466== Searching for pointers to 10,678 not-freed blocks
==21466== Checked 71,211,640 bytes
==21466== 
==21466== 984 bytes in 42 blocks are still reachable in loss record 1 of 6
==21466==    at 0x4845494: operator new(unsigned long, std::nothrow_t const&) (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==21466== 
==21466== 2,560 bytes in 8 blocks are possibly lost in loss record 2 of 6
==21466==    at 0x4846B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==21466== 
==21466== 17,512 bytes in 3 blocks are still reachable in loss record 3 of 6
==21466==    at 0x4846D10: realloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==21466== 
==21466== 405,564 bytes in 977 blocks are still reachable in loss record 4 of 6
==21466==    at 0x4846B0C: calloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==21466== 
==21466== 468,429 bytes in 3,965 blocks are still reachable in loss record 5 of 6
==21466==    at 0x4844BFC: malloc (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==21466== 
==21466== 2,166,764 bytes in 5,683 blocks are still reachable in loss record 6 of 6
==21466==    at 0x484522C: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-arm64-linux.so)
==21466== 
==21466== LEAK SUMMARY:
==21466==    definitely lost: 0 bytes in 0 blocks
==21466==    indirectly lost: 0 bytes in 0 blocks
==21466==      possibly lost: 2,560 bytes in 8 blocks
==21466==    still reachable: 3,059,253 bytes in 10,670 blocks
==21466==         suppressed: 0 bytes in 0 blocks

Is there anything else to try to make valgrind provide more info on leak source?

ENV

  • hw pack: NVIDIA Jetson
  • nv_tegra_release: R32 (release), REVISION: 5.1, GCID: 27362550, BOARD: t186ref, EABI: aarch64, DATE: Wed May 19 18:16:00 UTC 2021
  • linux distro: Ubuntu 18.04.5 LTS
  • cpu model: AMD Ryzen 5 3600X 6-Core Processor
  • cpu arch: x86_64
  • valgrind: 3.13.0

References

UPDATE

Uninstalling valgrind (v3.13) previously installed via apt and installing valgrind (v3.17) via snap solved the issue.


Solution

  • In case of problems with valgrind, it is always recommended to try with a recent version, either the last release or the git version.

    Note that it is quite easy to recompile valgrind from sources, at it has very few dependencies.

    In case of specific problems with stack traces, it is always useful to compare the stack traces produced by valgrind and gdb by using valgrind + gdb + vgdb.

    Put a breakpoint in gdb at relevant places, and you can then compare the gdb stacktrace produced by the gdb backtrace command and the valgrind stacktrace produced by the monitoring command:

    (gdb)  monitor v.info scheduler