I'm getting a sporadic segfault in my Rails app. It might be a problem coming from Sidekiq since they've been occurring randomly when jobs are being processed, but there's not much else to go on since the segfault is originating in libc/libcurl
Here's the backtrace that we're seeing:
/lib/libc.so.6(__fortify_fail+0x37)[0x7f225cde1357]
/lib/libc.so.6(+0x1042e9)[0x7f225cde12e9]
/lib/libc.so.6(__longjmp_chk+0x33)[0x7f225cde1253]
/usr/lib/libcurl.so(+0x91e5)[0x7f224bdc11e5]
/lib/libpthread.so.0(+0xf8f0)[0x7f225d93d8f0]
/lib/libc.so.6(__select+0x33)[0x7f225cdc1113]
sidekiq 3.2.1 app [15 of 15 busy](rb_thread_fd_select+0x49e)[0x7f225e131ebe]
sidekiq 3.2.1 app [15 of 15 busy](+0x5ebc0)[0x7f225e017bc0]
sidekiq 3.2.1 app [15 of 15 busy](rb_ensure+0x13c)[0x7f225dfde9bc]
sidekiq 3.2.1 app [15 of 15 busy](+0x4dcb9)[0x7f225e006cb9]
sidekiq 3.2.1 app [15 of 15 busy](+0x15130a)[0x7f225e10a30a]
sidekiq 3.2.1 app [15 of 15 busy](+0x166d7b)[0x7f225e11fd7b]
sidekiq 3.2.1 app [15 of 15 busy](+0x15cdd1)[0x7f225e115dd1]
sidekiq 3.2.1 app [15 of 15 busy](+0x1622fc)[0x7f225e11b2fc]
sidekiq 3.2.1 app [15 of 15 busy](rb_iseq_eval+0x16e)[0x7f225e11c1ce]
sidekiq 3.2.1 app [15 of 15 busy](+0x1cde1f)[0x7f225e186e1f]
sidekiq 3.2.1 app [15 of 15 busy](+0x1cdef6)[0x7f225e186ef6]
sidekiq 3.2.1 app [15 of 15 busy](+0x15130a)[0x7f225e10a30a]
sidekiq 3.2.1 app [15 of 15 busy](+0x166d7b)[0x7f225e11fd7b]
sidekiq 3.2.1 app [15 of 15 busy](+0x15cdd1)[0x7f225e115dd1]
sidekiq 3.2.1 app [15 of 15 busy](+0x1622fc)[0x7f225e11b2fc]
sidekiq 3.2.1 app [15 of 15 busy](rb_iseq_eval_main+0x300)[0x7f225e11b920]
sidekiq 3.2.1 app [15 of 15 busy](+0x262ea)[0x7f225dfdf2ea]
sidekiq 3.2.1 app [15 of 15 busy](ruby_run_node+0x36)[0x7f225dfe1086]
sidekiq 3.2.1 app [15 of 15 busy](+0x247db)[0x7f225dfdd7db]
/lib/libc.so.6(__libc_start_main+0xfd)[0x7f225ccfbcad]
sidekiq 3.2.1 app [15 of 15 busy](+0x246b9)[0x7f225dfdd6b9]
I'm not sure how to go about fixing this, since the output isn't super helpful and it isn't originating from any one bit of Ruby/Rails code. Any pointers or ideas on how to debug or what might actually be causing this? A lot of our jobs rely on curl to call third party APIs, so that might be a starting point.
Edit: I'm also seeing this in the logs:
*** longjmp causes uninitialized stack frame ***
I'd bet you are using the Typhoeus gem? Don't use it. https://github.com/mperham/sidekiq/issues/1400