I am using Jetson Xavier NX. I rewrote config.toml as follows and restarted the container However, when I execute describe to gpu pod, it seems to be warning. The pod is in a pending state all the time.
Edge Device: Jetson Xavier NX（control-plane and worker node node1） kubernetes plugin: k8s-device-plugin
Construction method: kuberspray Runtime: containerd
■ sudo nerdctl run --rm --gpus all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
FATA[0008] failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: src: /usr/lib/aarch64-linux-gnu/libcudnn.so.8, src_lnk: libcudnn.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn.so.8, dst_lnk: libcudnn.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn.so, src_lnk: /etc/alternatives/libcudnn_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn.so, dst_lnk: /etc/alternatives/libcudnn_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8, src_lnk: libcudnn_ops_infer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8, dst_lnk: libcudnn_ops_infer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so, src_lnk: /etc/alternatives/libcudnn_ops_infer_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so, dst_lnk: /etc/alternatives/libcudnn_ops_infer_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8, src_lnk: libcudnn_ops_train.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8, dst_lnk: libcudnn_ops_train.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so, src_lnk: /etc/alternatives/libcudnn_ops_train_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so, dst_lnk: /etc/alternatives/libcudnn_ops_train_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8, src_lnk: libcudnn_adv_infer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8, dst_lnk: libcudnn_adv_infer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so, src_lnk: /etc/alternatives/libcudnn_adv_infer_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so, dst_lnk: /etc/alternatives/libcudnn_adv_infer_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8, src_lnk: libcudnn_cnn_infer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8, dst_lnk: libcudnn_cnn_infer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so, src_lnk: /etc/alternatives/libcudnn_cnn_infer_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so, dst_lnk: /etc/alternatives/libcudnn_cnn_infer_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8, src_lnk: libcudnn_adv_train.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8, dst_lnk: libcudnn_adv_train.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so, src_lnk: /etc/alternatives/libcudnn_adv_train_so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so, dst_lnk: /etc/alternatives/libcudnn_adv_train_so
src: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8, src_lnk: libcudnn_cnn_train.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8, dst_lnk: libcudnn_cnn_train.so.8.2.1
src: /usr/include/cudnn_adv_infer.h, src_lnk: /etc/alternatives/cudnn_adv_infer_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_adv_infer.h, dst_lnk: /etc/alternatives/cudnn_adv_infer_h
src: /usr/include/cudnn_adv_train.h, src_lnk: /etc/alternatives/cudnn_adv_train_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_adv_train.h, dst_lnk: /etc/alternatives/cudnn_adv_train_h
src: /usr/include/cudnn_backend.h, src_lnk: /etc/alternatives/cudnn_backend_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_backend.h, dst_lnk: /etc/alternatives/cudnn_backend_h
src: /usr/include/cudnn_cnn_infer.h, src_lnk: /etc/alternatives/cudnn_cnn_infer_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_cnn_infer.h, dst_lnk: /etc/alternatives/cudnn_cnn_infer_h
src: /usr/include/cudnn_cnn_train.h, src_lnk: /etc/alternatives/cudnn_cnn_train_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_cnn_train.h, dst_lnk: /etc/alternatives/cudnn_cnn_train_h
src: /usr/include/cudnn.h, src_lnk: /etc/alternatives/libcudnn, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn.h, dst_lnk: /etc/alternatives/libcudnn
src: /usr/include/cudnn_ops_infer.h, src_lnk: /etc/alternatives/cudnn_ops_infer_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_ops_infer.h, dst_lnk: /etc/alternatives/cudnn_ops_infer_h
src: /usr/include/cudnn_ops_train.h, src_lnk: /etc/alternatives/cudnn_ops_train_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_ops_train.h, dst_lnk: /etc/alternatives/cudnn_ops_train_h
src: /usr/include/cudnn_version.h, src_lnk: /etc/alternatives/cudnn_version_h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/include/cudnn_version.h, dst_lnk: /etc/alternatives/cudnn_version_h
src: /etc/alternatives/libcudnn, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_v8.h
src: /etc/alternatives/libcudnn_adv_infer_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_adv_infer_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_infer.so.8
src: /etc/alternatives/libcudnn_adv_train_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_adv_train_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_adv_train.so.8
src: /etc/alternatives/libcudnn_cnn_infer_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_cnn_infer_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_infer.so.8
src: /etc/alternatives/libcudnn_cnn_train_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_cnn_train_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_cnn_train.so.8
src: /etc/alternatives/libcudnn_ops_infer_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_ops_infer_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_infer.so.8
src: /etc/alternatives/libcudnn_ops_train_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_ops_train_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn_ops_train.so.8
src: /etc/alternatives/libcudnn_so, src_lnk: /usr/lib/aarch64-linux-gnu/libcudnn.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/libcudnn_so, dst_lnk: /usr/lib/aarch64-linux-gnu/libcudnn.so.8
src: /etc/alternatives/cudnn_adv_infer_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_infer_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_adv_infer_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_infer_v8.h
src: /etc/alternatives/cudnn_backend_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_backend_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_backend_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_backend_v8.h
src: /etc/alternatives/cudnn_cnn_train_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_train_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_cnn_train_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_train_v8.h
src: /etc/alternatives/cudnn_ops_train_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_train_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_ops_train_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_train_v8.h
src: /etc/alternatives/cudnn_adv_train_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_train_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_adv_train_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_adv_train_v8.h
src: /etc/alternatives/cudnn_cnn_infer_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_infer_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_cnn_infer_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_cnn_infer_v8.h
src: /etc/alternatives/cudnn_ops_infer_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_infer_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_ops_infer_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_ops_infer_v8.h
src: /etc/alternatives/cudnn_version_h, src_lnk: /usr/include/aarch64-linux-gnu/cudnn_version_v8.h, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/alternatives/cudnn_version_h, dst_lnk: /usr/include/aarch64-linux-gnu/cudnn_version_v8.h
src: /usr/lib/aarch64-linux-gnu/libcudnn_static.a, src_lnk: /etc/alternatives/libcudnn_stlib, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcudnn_static.a, dst_lnk: /etc/alternatives/libcudnn_stlib
src: /usr/lib/aarch64-linux-gnu/libnvinfer.so.8, src_lnk: libnvinfer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer.so.8, dst_lnk: libnvinfer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8, src_lnk: libnvinfer_plugin.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so.8, dst_lnk: libnvinfer_plugin.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvparsers.so.8, src_lnk: libnvparsers.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvparsers.so.8, dst_lnk: libnvparsers.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvonnxparser.so.8, src_lnk: libnvonnxparser.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvonnxparser.so.8, dst_lnk: libnvonnxparser.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvinfer.so, src_lnk: libnvinfer.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer.so, dst_lnk: libnvinfer.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so, src_lnk: libnvinfer_plugin.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvinfer_plugin.so, dst_lnk: libnvinfer_plugin.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvparsers.so, src_lnk: libnvparsers.so.8.2.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvparsers.so, dst_lnk: libnvparsers.so.8.2.1
src: /usr/lib/aarch64-linux-gnu/libnvonnxparser.so, src_lnk: libnvonnxparser.so.8, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libnvonnxparser.so, dst_lnk: libnvonnxparser.so.8
src: /etc/vulkan/icd.d/nvidia_icd.json, src_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/etc/vulkan/icd.d/nvidia_icd.json, dst_lnk: /usr/lib/aarch64-linux-gnu/tegra/nvidia_icd.json
src: /usr/lib/aarch64-linux-gnu/libcuda.so, src_lnk: tegra/libcuda.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libcuda.so, dst_lnk: tegra/libcuda.so
src: /usr/lib/aarch64-linux-gnu/libdrm_nvdc.so, src_lnk: tegra/libdrm.so.2, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libdrm_nvdc.so, dst_lnk: tegra/libdrm.so.2
src: /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, src_lnk: tegra/libnvv4l2.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999, dst_lnk: tegra/libnvv4l2.so
src: /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, src_lnk: tegra/libnvv4lconvert.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4lconvert.so.0.0.999999, dst_lnk: tegra/libnvv4lconvert.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, src_lnk: ../../../tegra/libv4l2_nvargus.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvargus.so, dst_lnk: ../../../tegra/libv4l2_nvargus.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvcuvidvideocodec.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvcuvidvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvcuvidvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvidconv.so, src_lnk: ../../../tegra/libv4l2_nvvidconv.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvidconv.so, dst_lnk: ../../../tegra/libv4l2_nvvidconv.so
src: /usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, src_lnk: ../../../tegra/libv4l2_nvvideocodec.so, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libv4l/plugins/nv/libv4l2_nvvideocodec.so, dst_lnk: ../../../tegra/libv4l2_nvvideocodec.so
src: /usr/lib/aarch64-linux-gnu/libvulkan.so.1.2.141, src_lnk: tegra/libvulkan.so.1.2.141, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/libvulkan.so.1.2.141, dst_lnk: tegra/libvulkan.so.1.2.141
src: /usr/lib/aarch64-linux-gnu/tegra/libcuda.so, src_lnk: libcuda.so.1.1, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libcuda.so, dst_lnk: libcuda.so.1.1
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, src_lnk: libnvbufsurface.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so, dst_lnk: libnvbufsurface.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, src_lnk: libnvbufsurftransform.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvbufsurftransform.so, dst_lnk: libnvbufsurftransform.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, src_lnk: libnvbuf_utils.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so, dst_lnk: libnvbuf_utils.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, src_lnk: libnvdsbufferpool.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvdsbufferpool.so, dst_lnk: libnvdsbufferpool.so.1.0.0
src: /usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, src_lnk: libnvid_mapper.so.1.0.0, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/aarch64-linux-gnu/tegra/libnvid_mapper.so, dst_lnk: libnvid_mapper.so.1.0.0
src: /usr/share/glvnd/egl_vendor.d/10_nvidia.json, src_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/share/glvnd/egl_vendor.d/10_nvidia.json, dst_lnk: ../../../lib/aarch64-linux-gnu/tegra-egl/nvidia.json
src: /usr/lib/libvisionworks_sfm.so, src_lnk: libvisionworks_sfm.so.0.90, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_sfm.so, dst_lnk: libvisionworks_sfm.so.0.90
src: /usr/lib/libvisionworks_sfm.so.0.90, src_lnk: libvisionworks_sfm.so.0.90.4, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_sfm.so.0.90, dst_lnk: libvisionworks_sfm.so.0.90.4
src: /usr/lib/libvisionworks.so, src_lnk: libvisionworks.so.1.6, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks.so, dst_lnk: libvisionworks.so.1.6
src: /usr/lib/libvisionworks_tracking.so, src_lnk: libvisionworks_tracking.so.0.88, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_tracking.so, dst_lnk: libvisionworks_tracking.so.0.88
src: /usr/lib/libvisionworks_tracking.so.0.88, src_lnk: libvisionworks_tracking.so.0.88.2, dst: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/lib/libvisionworks_tracking.so.0.88, dst_lnk: libvisionworks_tracking.so.0.88.2
, stderr: nvidia-container-cli: mount error: stat failed: /run/containerd/io.containerd.runtime.v2.task/k8s.io/9002182b05c0a2464ef1eb5081d73742f5862a3e7fea321fbc5e3410f3f3b602/rootfs/usr/local/cuda-10.2/compat/libcuda.so.440.118.02: no such file or directory: unknown
``
To use GPUs in Kubernetes, the NVIDIA Device Plugin is required. The NVIDIA Device Plugin is a daemonset that automatically enumerates the number of GPUs on each node of the cluster and allows pods to be run on GPUs. find this link to install.
As you are getting the error failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook
ADD below in YAML and it works .
FROM node:16-alpine
ENV NODE_ENV="development"
WORKDIR /app
COPY package.json .
COPY package-lock.json .
ARG NODE_ENV
RUN apk add g++ make py3-pip
RUN npm install
RUN chown -R node /app/node_modules
RUN npm install -g ts-node nodemon
COPY . ./
ENV PORT 8000
EXPOSE $PORT
CMD ["ts-node", "./src/server.ts"]
Refer this SO which is having the same error sorted by using
RUN chown -R node /app/node_modules.This issue was Linux specific.