Why does docker and docker compose gives separate results while accessing /opt directory?

Question

I have a docker compose file that contains 2 containers, the completion of the first container is essential before the second container starts.

If I run the following docker command: docker run -v /opt/kubearmor/BPF:/opt/kubearmor/BPF --privileged kubearmor/kubearmor-init:stable it will compile bpf code for the current kernel and output the file as /opt/kubearmor/BPF/system_monitor.bpf.o. This file will then used by the second container.

If I run the above docker command after adding necessary read/write permissions to the /opt/kubearmor/ directory, it runs successfully and outputs the file.

But if I run it as part of the following docker-compose.yaml file, the first container isn't able to create that system_monitor.bpf.o file at the desired directory.

docker-compose.yaml

services:
  kubearmor-init:
    image: kubearmor/kubearmor-init:stable
    privileged: true
    volumes:
      - /opt/kubearmor/BPF:/opt/kubearmor/BPF
    entrypoint: ["/bin/sh", "-c", "echo 'Init container done' && sleep 10"]

  kubearmor:
    image: kubearmor/kubearmor:latest
    command: ["-k8s=false"]
    depends_on:
      kubearmor-init:
        condition: service_completed_successfully
    privileged: true
    pid: "host"
    ipc: "host"
    network_mode: "host"
    volumes:
      - /opt/kubearmor/BPF:/opt/kubearmor/BPF
      - /sys/fs/bpf:/sys/fs/bpf
      - /sys/kernel/security:/sys/kernel/security
      - /sys/kernel/debug:/sys/kernel/debug
      - /var/run/containerd/containerd.sock:/var/run/containerd/containerd.sock
      - /run/containerd:/run/containerd
      - /var/lib/docker:/var/lib/docker

As a result the second container gives the error:

kubearmor-1       | 2024-05-20 15:18:52.494014  ERROR   Failed to initialize BPF (cannot load bpf module specs open /opt/kubearmor/BPF/system_monitor.bpf.o: no such file or directory)

I tried giving full read/write access to the opt/kubearmor/BPF directory but it works for the standalone docker command only and not for the docker-compose method.

Answer 1

It appears that you're overriding the kubearmor-init image's default entrypoint when you use Compose.

The image uses: ENTRYPOINT ["/KubeArmor/compile.sh"]

... and your service is using: ENTRYPOINT ["/bin/sh", "-c", "echo 'Init container done' && sleep 10"], which overrides the default and does not output the file expected by the kubearmor service.

If you change entrypoint to command in your kubearmor-init service definition or add a call to that script before your echo, the file should be output to /opt/... as expected.