Search code examples
dpdk

Running a DPDK application binary with all dependent library (compiled in different machine) on another machine

  • A DPDK application with several runtime dependent libraries are compiled on one machine

  • Binaries and libraries are copied from that machine to another machine with similar specs and environment

  • Running the DPDK application with the parameters as given below, but the application crashes during rte_eal_init()

 App-binary -l 1 -a 0000:02:00.0 -a 0000:03:00.0 -d /opt/upf/lib/ --proc-type=primary --file-prefix=.app_0000:02:00.0
  • This is the back trace from gnu debugger crash core file
    #0  0x00007faaa0ead337 in raise () from /lib64/libc.so.6
    #1  0x00007faaa0eaea28 in abort () from /lib64/libc.so.6
    #2  0x00007faaa125104f in __rte_panic () from /opt/upf/lib/librte_eal.so.21
    #3  0x00007faa9e228e1c in tailqinitfn_rte_ring_tailq () from /opt/upf/lib/librte_ring.so.21.0
    #4  0x00007faaa278a973 in _dl_init_internal () from /lib64/ld-linux-x86-64.so.2
    #5  0x00007faaa278f54e in dl_open_worker () from /lib64/ld-linux-x86-64.so.2
    #6  0x00007faaa278a784 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
    #7  0x00007faaa278eb3b in _dl_open () from /lib64/ld-linux-x86-64.so.2
    #8  0x00007faaa0c73eeb in dlopen_doit () from /lib64/libdl.so.2
    #9  0x00007faaa278a784 in _dl_catch_error () from /lib64/ld-linux-x86-64.so.2
    #10 0x00007faaa0c744ed in _dlerror_run () from /lib64/libdl.so.2
    #11 0x00007faaa0c73f81 in dlopen@@GLIBC_2.2.5 () from /lib64/libdl.so.2
    #12 0x00007faaa125bc55 in eal_plugins_init () from /opt/upf/lib/librte_eal.so.21
    #13 0x00007faaa126f2ba in rte_eal_init () from /opt/upf/lib/librte_eal.so.21
    #14 0x000000000041414a in Dpdk_LibTask (arg=<optimized out>) at /root/5g_upf/core/service/common/dpdk/dpdk.c:1244
    #15 0x00007faaa2566e65 in start_thread () from /lib64/libpthread.so.0
    #16 0x00007faaa0f7588d in clone () from /lib64/libc.so.6

Updates:

  • Host Machine details:
    • i3 8100 3.6GHz 4 Cores
    • 8 GB RAM
    • CentOS 7
    • gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
    • GNU ld version 2.27-44.base.el7
    • DPDK 20.11.0
    • 3 NICs bounded to DPDK 
      0000:01:00.0 '82574L Gigabit Network Connection 10d3' drv=uio_pci_generic unused=e1000e
0000:07:00.0 '82574L Gigabit Network Connection 10d3' drv=uio_pci_generic unused=e1000e
0000:08:00.0 '82574L Gigabit Network Connection 10d3' drv=uio_pci_generic unused=e1000e
  • Target Machine details:
    • i3 8100 3.6GHz 4 Cores
    • 8 GB RAM
    • CentOS 7
    • gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC)
    • GNU ld version 2.27-44.base.el7
    • DPDK 20.11.0
    • 3 NICs bounded to DPDK 
      0000:02:00.0 '82574L Gigabit Network Connection 10d3' drv=uio_pci_generic unused=e1000e
0000:03:00.0 '82574L Gigabit Network Connection 10d3' drv=uio_pci_generic unused=e1000e
0000:04:00.0 '82574L Gigabit Network Connection 10d3' drv=uio_pci_generic unused=e1000e
Solution

  • [Based on the live debug with Sumesh].

    Application background:

    1. The application has dependency on DPDK libraries, 3rd party libraries and GNU libraries.
    2. Actual open source project, builds DPDK 20.11.
    3. Docker instance is started with docker run where root permission is shared and and copies these over to a local folder (same machine).
    4. With LD_LIBRARY_PATH set to desired folder DPDK libraries dependency are corrected.

    What caused the issues:

    1. Machine-A is used to build the DPDK 20.11 libraries.
    2. Instead of running docker instance on Machine-A, Machine-B is choose as target machine.
    3. DPDK libraries are copied from Machine-A to Machine-B.
    4. Docker-run is used to start the application in Machine-B

    How to fix the issue:

    1. DPDK libraries when built has numerous other libraries and version dependency
    2. Build and install DPDK on target (MACHINE-B) and do not copy.
    3. in docker-run share the permission to access the /usr/lib/lib64 which houses DPDK, 3rd party and GNU Libraries.
    4. update the LD_LIBRARY_PATH to have access to right folder to resolve the dependency.

    Note:

    1. Tested and validated on both Host and Docker with hello-world for sanity
    2. Sumesh is updating the scripts to reflect the folder permission for the custom application.