Search code examples
pythonlinuxoperating-systemsystem-callsstrace

strace: What do consecutive <unfinished...> syscall with no <resumed> mean?


I'm trying to debug a multithreaded network server written in Python with strace -tt -f, and the strace output makes me confused because there appear to be multiple consecutive syscalls with <unfinished...> on the same thread, and no resumed is observed after that:

9675  22:58:28.407844 <... recvfrom resumed> "\22RT\21Q\0\n", 7, 0, NULL, NULL) = 7
9675  22:58:28.518079 recvfrom(3, "\22RV\21Q\3\n", 7, 0, NULL, NULL) = 7
9675  22:58:28.518672 sendto(7, "\0\0\0\6\0\22RV\21Q\0\n", 12, 0, NULL, 0) = 12
9675  22:58:28.536676 poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0 <unfinished ...>
9675  22:58:28.536739 <... poll resumed> ) = 1 ([{fd=7, revents=POLLIN}])
9675  22:58:28.538515 recvfrom(7,  <unfinished ...>
9675  22:58:28.539400 recvfrom(7,  <unfinished ...>
9675  22:58:28.539768 sendto(7, "\0\0\0\6\0\22RW\21Q\0\n", 12, 0, NULL, 0 <unfinished ...>
9675  22:58:28.540629 poll([{fd=7, events=POLLIN|POLLPRI}], 1, 0 <unfinished ...>
9675  22:58:28.540666 <... poll resumed> ) = 1 ([{fd=7, revents=POLLIN|POLLERR|POLLHUP}])
9675  22:58:28.541170 recvfrom(7,  <unfinished ...>
9675  22:58:28.542603 recvfrom(7,  <unfinished ...>
9675  22:58:28.543449 shutdown(7, SHUT_RDWR <unfinished ...>
9675  22:58:28.544451 close(7 <unfinished ...>
9656  22:58:28.555189 accept4(5, {sa_family=AF_INET, sin_port=htons(38282), sin_addr=inet_addr("127.0.0.1")}, [16], SOCK_CLOEXEC) = 7
9656  22:58:28.555288 ioctl(7, FIONBIO, [0]) = 0
9656  22:58:28.555338 ioctl(7, FIONBIO, [0]) = 0
9656  22:58:28.555749 getpeername(7, {sa_family=AF_INET, sin_port=htons(38282), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
9656  22:58:28.555810 getsockname(7, {sa_family=AF_INET, sin_port=htons(6000), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
9656  22:58:28.555929 getpeername(7, {sa_family=AF_INET, sin_port=htons(38282), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
9675  22:58:28.560799 sendto(7, "\0\0\0\10\0\22QP\21R\21Q\2\n", 14, 0, NULL, 0) = 14

Solution

  • According to @OznOg's suggestion, it's because the multi-threaded application is running too fast thus strace drops some results.

    As a better alternative, I switched to the faster perf trace ./app and problem was solved. Note that if you see perf trace complaining ignored trace, you can simply increase event buffer size by sudo sh -c "echo 1032 > /proc/sys/kernel/perf_event_mlock_kb"