error mounting "/sys/kernel/debug/tracing" to rootfs

Question

I'm using ebpf in container environment to trace several kernel tracepoints.

Part of the code is shown beloew:

SEC("tp/sched/sched_wakeup")
int handle__sched_wakeup(struct sched_wakeup_tp_args *ctx)
{
    struct task_struct *task = (void *)bpf_get_current_task();

   return some_function(task);
}

This is quite a simple ebpf program and can be run successfully on host machine. When I warp this into container, and load it into kernel using cilium/ebpf like this:

// Load pre-compiled programs and maps into the kernel.
objs := bpfObjects{}
if err := loadBpfObjects(&objs, nil); err != nil {
    return nil, fmt.Errorf("load bpf objects error: %v", err)
}

I get an error as :

failed to load eBPF program : link tracepoint sched_wakeup error: trace event sched/sched_wakeup: file does not exist

I check the filepath in my container /sys/kernel/debug and it shows nothing. While in my host machine it has the tracing sub-directory. And my mount | grep kernel shows:

securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)

If I try to use volume mount in my container, the container reports an error and cannot be started:

Error: failed to create containerd task: failed to create shim: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/sys/kernel/debug/tracing" to rootfs at "/sys/kernel/debug/tracing": mkdir /run/containerd/io.containerd.runtime.v2.task/k8s.io/mycontainername/rootfs/sys/kernel/debug/tracing: no such file or directory: unknown

So is there any way to solve this problem? Why is this error happening and how to let my container use tracepoints? Some related linux knowledge or container knowledge might help if you can provide. Thanks a lot!

Answer 1

Your container lives in its own isolated file system, when it attempts to access /sys/fs/debug/tracing it is actually accessing /run/containerd/io.containerd.runtime.v2.task/k8s.io/mycontainername/rootfs/sys/kernel/debug/tracing on your host, not the actual /sys/fs/debug/tracing where your tracingFS is mounted. This is a security feature.

So you need to tell your container engine to map the path inside the container to the same path outside the container via a mount/volume -v /sys/:/sys/ that should solve your problem. However, bidirectional propagation (write access to the host) might require the container to be priviliged.