Search code examples
cclangllvmfuzzingaddress-sanitizer

LLVM libFuzzer rss memory increase

I'm experimenting with libFuzzer of the LLVM project trying to generate inputs for an audio signal processing library.

For the sake of simplicity I started with a constant-zero 64kb PCM as initial corpus, using -max_len=65536. Memory consumption per library call is about 200kB, which should be freed as expected.

What I'm wondering about is the memory consumption shown as rss in the fuzzer log:

#1024   pulse  cov: 26153 ft: 40811 corp: 783/33Mb exec/s: 7 rss: 541Mb
#2048   pulse  cov: 26154 ft: 40811 corp: 783/33Mb exec/s: 8 rss: 573Mb
#4096   pulse  cov: 26154 ft: 40811 corp: 783/33Mb exec/s: 8 rss: 574Mb
#4895   NEW    cov: 26156 ft: 40815 corp: 784/33Mb exec/s: 8 rss: 576Mb L: 26935 MS: 1 InsertRepeatedBytes-
#8192   pulse  cov: 26157 ft: 40815 corp: 784/33Mb exec/s: 8 rss: 578Mb
#11875  NEW    cov: 26157 ft: 40816 corp: 785/33Mb exec/s: 8 rss: 579Mb L: 59394 MS: 1 InsertByte-
#12862  NEW    cov: 26157 ft: 40817 corp: 786/33Mb exec/s: 8 rss: 579Mb L: 51404 MS: 3 EraseBytes-EraseBytes-CopyPart-
#15357  NEW    cov: 26157 ft: 40818 corp: 787/33Mb exec/s: 8 rss: 579Mb L: 64536 MS: 3 CopyPart-ChangeBit-InsertRepeatedBytes-
#16384  pulse  cov: 26157 ft: 40818 corp: 787/33Mb exec/s: 8 rss: 579Mb
#23171  NEW    cov: 26157 ft: 40820 corp: 788/33Mb exec/s: 8 rss: 579Mb L: 49412 MS: 2 InsertRepeatedBytes-CMP- DE: "\x00\x00\x00\x00\x00\x00\x00\x00"-
#25426  NEW    cov: 26158 ft: 40825 corp: 789/33Mb exec/s: 8 rss: 579Mb L: 64443 MS: 2 CMP-CMP- DE: "\x00\x00\x00\x00"-"\xff\xff"-

Why is the rss metric increasing steadily?

Note that the increase seems to converge (similar to the cov and ft quantities), so memory isn't leaked per fuzzer target invocation.

Is it a setup problem or a leak, which causes rss memory to increase?

If not, what is the rationale of having an -rss_limit_mb option available to the fuzzer, if the memory consumption isn't set back after each fuzzer target invocation?

Solution

  • Some reasons for memory increase are:

    • ASan's "quarantine": ASan keeps freed memory in quarantine to detect use-after-free errors. When fuzzing without a quarantine size limit, memory growth is unbounded. Try to change the quarantine_size_mb ASan option to a different value than -1 which means unlimited. ASan should limit the quarantine size by default these days.

    • Internal data structures of libFuzzer but these should stop growing at some point.

    • Memory allocated by your code that isn't returned to the OS after being freed. The RSS value will often reflect the maximum used by any of the test inputs and won't go down after that. All freed memory is available to subsequent allocations, of course.

    • Another explanation is that your code really has a memory leak. Many leaks are detected with the detect_leaks option which is on by default. But it's also possible that a global data structure like a cache is growing constantly which can't be detected when fuzzing.

    With the quarantine_size_mb option, RSS should converge eventually.