Search code examples
phpmultithreadingredispthreadspredis

PHP + PThreads + Redis/Predis = zend_mm_heap corrupted?

I have been struggling with a mysterious error for a few days now. I am using PHP 7.1.0RC3 (I recompiled myself with ZTS/pthreads enabled). Recently, I have been working on a refactor that replaces MySQL with Redis to optimize non-disk-worthy data I/O in my application.

I have a script that creates a Thread (A) for each Cryptocurrency Market. Thread (A) creates another Thread (B) for each Trading Strategy. B Threads are always synced before A Threads.

I keep getting this error: zend_mm_heap corrupted. It occurs at a different point in execution each time I run the script.

I have tried all suggested fixes, 100s of Google pages. Garbage Collection, PHP configuration/compilation, all has been reviewed to great detail. I haven't found anything related to "zend_mm_heap corrupted", PThreads and Redis/Predis, while other "fixes" have had no effect...

All data is read from and saved to Redis (via Predis). The error message only appears when using Predis within Thread A or Thread B (changes each time). It occurs more frequently when two Predis method invocations happen in rapid succession (set then get, for example).

If I use shell_exec() to run Redis commands directly via the CLI, I don't see this error. But, it's significantly slower so may simply be missing this edge case?

If anyone can give any input/thoughts towards any of the following, I would greatly appreciate it:

  1. How can I find where this is coming from in my application?
  2. Would a dedicated Redis Server potentially fix this?
  3. Should I submit a bug report to https://github.com/krakjoe/pthreads?
  4. Should I submit a bug report to https://github.com/nrk/predis?
  5. Does anyone know why zend_mm_heap corrupted might occur, in general, in a thread "safe" environment (aka: ZTS)? Or how I should guard against it???
  6. Should I try compiling even more different versions of PHP 7+? https://github.com/php/php-src

I am out of options, please have mercy. I can share code samples but pasting them here would not be very useful.

Edit I recompiled PHP with ./configure --enable-maintainer-zts --enable-pthreads --with-pthreads-sanitize --with-mysqli --enable-embedded-mysqli and I saw a new error I haven't seen before:

==2716== ERROR: AddressSanitizer: heap-use-after-free on address 0x600800001c15 at pc 0xe25a87 bp 0x7f7a049feae0 sp 0x7f7a049fead0
READ of size 1 at 0x600800001c15 thread T1632
    #0 0xe25a86 (/usr/local/bin/php+0xe25a86)
    #1 0xe953c8 (/usr/local/bin/php+0xe953c8)
    #2 0xe4935b (/usr/local/bin/php+0xe4935b)
    #3 0xcbede3 (/usr/local/bin/php+0xcbede3)
    #4 0x98cfa9 (/usr/local/bin/php+0x98cfa9)
    #5 0x99dc3c (/usr/local/bin/php+0x99dc3c)
    #6 0x7f7b57e74a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
    #7 0x7f7b5694edc4 (/usr/lib64/libpthread-2.17.so+0x7dc4)
    #8 0x7f7b5667d73c (/usr/lib64/libc-2.17.so+0xf773c)
0x600800001c15 is located 5 bytes inside of 48-byte region [0x600800001c10,0x600800001c40)
freed by thread T1623 here:
    #0 0x7f7b57e71009 (/usr/lib64/libasan.so.0.0.0+0x16009)
    #1 0xe259fb (/usr/local/bin/php+0xe259fb)
    #2 0xe953c8 (/usr/local/bin/php+0xe953c8)
    #3 0xe4935b (/usr/local/bin/php+0xe4935b)
    #4 0xcbede3 (/usr/local/bin/php+0xcbede3)
    #5 0x98cfa9 (/usr/local/bin/php+0x98cfa9)
    #6 0x99dc3c (/usr/local/bin/php+0x99dc3c)
    #7 0x7f7b57e74a97 (/usr/lib64/libasan.so.0.0.0+0x19a97)
previously allocated by thread T0 here:
    #0 0x7f7b57e71129 (/usr/lib64/libasan.so.0.0.0+0x16129)
    #1 0xdb0118 (/usr/local/bin/php+0xdb0118)
    #2 0xe64f6c (/usr/local/bin/php+0xe64f6c)
    #3 0xe68c5d (/usr/local/bin/php+0xe68c5d)
    #4 0xcc4dcc (/usr/local/bin/php+0xcc4dcc)
    #5 0xcc5972 (/usr/local/bin/php+0xcc5972)
    #6 0x111f82c (/usr/local/bin/php+0x111f82c)
    #7 0x44c781 (/usr/local/bin/php+0x44c781)
    #8 0x7f7b565a7b34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T1632 created by T0 here:
    #0 0x7f7b57e65c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x99e4cb (/usr/local/bin/php+0x99e4cb)
    #2 0x975d2a (/usr/local/bin/php+0x975d2a)
    #3 0x11105c1 (/usr/local/bin/php+0x11105c1)
    #4 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #5 0x97b385 (/usr/local/bin/php+0x97b385)
    #6 0x11126bc (/usr/local/bin/php+0x11126bc)
    #7 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #8 0x97b385 (/usr/local/bin/php+0x97b385)
    #9 0x11126bc (/usr/local/bin/php+0x11126bc)
    #10 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #11 0x97b385 (/usr/local/bin/php+0x97b385)
    #12 0x11126bc (/usr/local/bin/php+0x11126bc)
    #13 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #14 0x97b385 (/usr/local/bin/php+0x97b385)
    #15 0x11126bc (/usr/local/bin/php+0x11126bc)
    #16 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #17 0x97b385 (/usr/local/bin/php+0x97b385)
    #18 0x1110b28 (/usr/local/bin/php+0x1110b28)
    #19 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #20 0x97b385 (/usr/local/bin/php+0x97b385)
    #21 0x11126bc (/usr/local/bin/php+0x11126bc)
    #22 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #23 0x97b385 (/usr/local/bin/php+0x97b385)
    #24 0x111c28a (/usr/local/bin/php+0x111c28a)
    #25 0xe52281 (/usr/local/bin/php+0xe52281)
    #26 0xcc766f (/usr/local/bin/php+0xcc766f)
    #27 0x1121d36 (/usr/local/bin/php+0x1121d36)
    #28 0x44d202 (/usr/local/bin/php+0x44d202)
    #29 0x7f7b565a7b34 (/usr/lib64/libc-2.17.so+0x21b34)
Thread T1623 created by T0 here:
    #0 0x7f7b57e65c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    #1 0x99e4cb (/usr/local/bin/php+0x99e4cb)
    #2 0x975d2a (/usr/local/bin/php+0x975d2a)
    #3 0x11105c1 (/usr/local/bin/php+0x11105c1)
    #4 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #5 0x97b385 (/usr/local/bin/php+0x97b385)
    #6 0x11126bc (/usr/local/bin/php+0x11126bc)
    #7 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #8 0x97b385 (/usr/local/bin/php+0x97b385)
    #9 0x11126bc (/usr/local/bin/php+0x11126bc)
    #10 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #11 0x97b385 (/usr/local/bin/php+0x97b385)
    #12 0x11126bc (/usr/local/bin/php+0x11126bc)
    #13 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #14 0x97b385 (/usr/local/bin/php+0x97b385)
    #15 0x11126bc (/usr/local/bin/php+0x11126bc)
    #16 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #17 0x97b385 (/usr/local/bin/php+0x97b385)
    #18 0x1110b28 (/usr/local/bin/php+0x1110b28)
    #19 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #20 0x97b385 (/usr/local/bin/php+0x97b385)
    #21 0x11126bc (/usr/local/bin/php+0x11126bc)
    #22 0xf71de2 (/usr/local/bin/php+0xf71de2)
    #23 0x97b385 (/usr/local/bin/php+0x97b385)
    #24 0x111c28a (/usr/local/bin/php+0x111c28a)
    #25 0xe52281 (/usr/local/bin/php+0xe52281)
    #26 0xcc766f (/usr/local/bin/php+0xcc766f)
    #27 0x1121d36 (/usr/local/bin/php+0x1121d36)
    #28 0x44d202 (/usr/local/bin/php+0x44d202)
    #29 0x7f7b565a7b34 (/usr/lib64/libc-2.17.so+0x21b34)
Shadow bytes around the buggy address:
  0x0c017fff8330: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
  0x0c017fff8340: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 fa
  0x0c017fff8350: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 00
  0x0c017fff8360: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
  0x0c017fff8370: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 00
=>0x0c017fff8380: fa fa[fd]fd fd fd fd fd fa fa 00 00 00 00 00 00
  0x0c017fff8390: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x0c017fff83a0: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x0c017fff83b0: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 00
  0x0c017fff83c0: fa fa 00 00 00 00 00 00 fa fa 00 00 00 00 00 fa
  0x0c017fff83d0: fa fa 00 00 00 00 00 fa fa fa 00 00 00 00 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:     fa
  Heap righ redzone:     fb
  Freed Heap region:     fd
  Stack left redzone:    f1
  Stack mid redzone:     f2
  Stack right redzone:   f3
  Stack partial redzone: f4
  Stack after return:    f5
  Stack use after scope: f8
  Global redzone:        f9
  Global init order:     f6
  Poisoned by user:      f7
  ASan internal:         fe
==2716== ABORTING
Solution

  • I found the source of the memory corruption, and the "fix" was very simple. I have found that the only "non-fatal" way of using a phpiredis $client in pthreads is to explicitly connect/execute commands/disconnect in the current local scope.

    Summed up, the following results in Memory Corruption when under high load:

    1. Attempting to pass a phpiredis $client to another object by reference in Threaded context
    2. Attempting to pass a phpiredis $client to another object by value in Threaded context
    3. Attempting to instantiate/use a phpiredis $client in anywhere except local scope in Threaded context (ex: in another class, _Redis, when we are currently in Investor::save)
    4. Attempting to statically reuse a phpiredis connection in general in Threaded context

    Memory corruption does not occur when passing a MySQLi $client to another object in the same way/context, though.

    I hope this saves someone else the week of despair I have witnessed. This is how one must use phpiredis within pthreads:

    // Instantiate a local raw phpiredis $client and execute get/set commands using $client directly
// @Result: This WORKS :D
$client = phpiredis_connect('127.0.0.1', 6379);
$key = "table_name-_-$this->id";
foreach($args as $prop=>$val)
    if(isset($prop) && isset($val))
        phpiredis_command($client, "hset $key $prop $val");
$obj = phpiredis_command($client, "hgetall $key");
echo json_encode($obj);
phpiredis_disconnect($client);