Search code examples
dockerubuntudocker-composenetdata

Docker containers running on the server are not visible in the netdata dashboard

ı installed netdata on my server and connected it to the netdata dashboard, but I noticed that I cannot see the Docker containers that are currently running on the server on the dashboard. To fix this, I updated the compose.yml file with the following volumes section:

     netdata:
    image: netdata/netdata
  ++pid:"host"
    container_name: netdata
    hostname: example.com # set to fqdn of host
    ports:
      - 19999:19999
    restart: unless-stopped
    cap_add:
      - SYS_PTRACE
    security_opt:
      - apparmor:unconfined
    volumes:
      - netdataconfig:/etc/netdata
      - netdatalib:/var/lib/netdata
      - netdatacache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      - /sys/fs/cgroup:/host/sys/fs/cgroup:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /user-service:/host/user-service:ro

and then I restarted the container. However, when I checked the logs of the netdata container, I saw the following error:

2023-01-05 10:08:38: cgroup-name.sh: INFO: docker container 

'7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e' is named 'notification-service'
2023-01-05 10:08:38: cgroup-name.sh: INFO: cgroup 'system.slice_docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope' is called 'notification-service'
2023-01-05 10:08:39: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : Using host prefix directory '/host'
2023-01-05 10:08:39: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open pid_from_cgroup() file '/host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope/tasks'. (errno 2, No such file or directory)
2023-01-05 10:08:39: /usr/libexec/netdata/plugins.d/cgroup-network INFO : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope'
2023-01-05 10:08:39: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1b89cc234ba1266278a09f61280f8b3bb707dd79f9fb46e.scope'

When I go to the bash of the netdata container, the file path /host/sys/fs/cgroup/system.slice/docker-7d6a282fefd31f1df1bs89cc234ba1sd266278a0s9f61280f8b3bb707dd79f9fb46e.scope actually exists. But there is no file named tasks in this file path. How can I solve this error and see every container running on the server on the dashboard?

I solved the above error by adding the pid line, but I still can't see the containers running on the server on the dashboard. netdata container logs as follows:

2023-01-05 12:43:35: cgroup-name.sh: INFO: cgroup 'init.scope' is called 'init.scope'
2023-01-05 12:43:35: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928/json
2023-01-05 12:43:35: cgroup-name.sh: INFO: docker container 'd1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928' is named 'definition-service'
2023-01-05 12:43:35: cgroup-name.sh: INFO: cgroup 'system.slice_docker-d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928.scope' is called 'definition-service'
2023-01-05 12:43:36: /usr/libexec/netdata/plugins.d/cgroup-network INFO  : MAIN : Using host prefix directory '/host'
2023-01-05 12:43:36: /usr/libexec/netdata/plugins.d/cgroup-network INFO  : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928.scope'
2023-01-05 12:43:37: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-d1e758ed14c0e33d53f1a6467daebb715db468818fb77e5cb1c194ad6a890928.scope'
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : child pid 1853221 exited with code 1.
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to network namespace of pid 1342633 (errno 1, Operation not permitted)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to pid namespace of pid 1342633 (errno 1, Operation not permitted)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to mount namespace of pid 1342633 (errno 1, Operation not permitted)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : PROCFILE: Cannot open file '/proc/net/dev' (errno 2, No such file or directory)
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open file '/proc/net/dev'
2023-01-05 12:43:37: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : cannot read cgroup interface list.
2023-01-05 12:43:37: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad/json
2023-01-05 12:43:37: cgroup-name.sh: INFO: docker container '459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad' is named 'mongo_db'
2023-01-05 12:43:37: cgroup-name.sh: INFO: cgroup 'system.slice_docker-459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad.scope' is called 'mongo_db'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network INFO  : MAIN : Using host prefix directory '/host'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network INFO  : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad.scope'
2023-01-05 12:43:38: ACLK STA [30c26177-2ef2-4768-b509-6e58e18e2643 (example.com)]: QUEUED REMOVED ALERTS
2023-01-05 12:43:38: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-459c9a2f6634443c1a0931d55628c60c6a268c9303052171cf4771ae88a340ad.scope'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : child pid 1853268 exited with code 1.
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to network namespace of pid 1343129 (errno 1, Operation not permitted)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to pid namespace of pid 1343129 (errno 1, Operation not permitted)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to mount namespace of pid 1343129 (errno 1, Operation not permitted)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : PROCFILE: Cannot open file '/proc/net/dev' (errno 2, No such file or directory)
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open file '/proc/net/dev'
2023-01-05 12:43:38: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : cannot read cgroup interface list.
2023-01-05 12:43:39: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179/json
2023-01-05 12:43:39: cgroup-name.sh: INFO: docker container 'ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179' is named 'mail-service'
2023-01-05 12:43:39: cgroup-name.sh: INFO: cgroup 'system.slice_docker-ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179.scope' is called 'mail-service'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network INFO  : MAIN : Using host prefix directory '/host'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network INFO  : MAIN : running: exec /usr/libexec/netdata/plugins.d/cgroup-network-helper.sh --cgroup '/host/sys/fs/cgroup/system.slice/docker-ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179.scope'
2023-01-05 12:43:40: cgroup-network-helper.sh: INFO: searching for network interfaces of cgroup '/host/sys/fs/cgroup/system.slice/docker-ee448a793745d2a72ed86fbcb26baa7de16f7103a816910684ad457b6557e179.scope'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : child pid 1853317 exited with code 1.
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to network namespace of pid 1499298 (errno 1, Operation not permitted)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to pid namespace of pid 1499298 (errno 1, Operation not permitted)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot switch to mount namespace of pid 1499298 (errno 1, Operation not permitted)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : PROCFILE: Cannot open file '/proc/net/dev' (errno 2, No such file or directory)
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : Cannot open file '/proc/net/dev'
2023-01-05 12:43:40: /usr/libexec/netdata/plugins.d/cgroup-network ERROR : MAIN : cannot read cgroup interface list.
2023-01-05 12:43:41: cgroup-name.sh: INFO: Running API command: curl --unix-socket "/var/run/docker.sock" http://localhost/containers/bcb761b2cd1e89d530cc8973ed21a894e21bc50d0d7138eb8605705ba4ea7b32/json
2023-01-05 12:43:41: cgroup-name.sh: INFO: docker container 'bcb761b2cd1e89d530cc8973ed21a894e21bc50d0d7138eb8605705ba4ea7b32' is named 'netdata'
2023-01-05 12:43:41: cgroup-name.sh: INFO: cgroup 'system.slice_docker-bcb761b2cd1e89d530cc8973ed21a894e21bc50d0d7138eb8605705ba4ea7b32.scope' is called 'netdata'
Solution

  • I saw this similar issue https://github.com/netdata/netdata/issues/11069 and the fix seemed to be to add pid: "host", check this comment

    https://github.com/netdata/netdata/issues/11069#issuecomment-952228202

    Can you try and see if it works?