Search code examples
docker-composekvmrocky-os

Rocky Linux 8 with KVM and docker - system halt after some time

I installed Rocky linux 8 with KVM and docker support. I configured bridge networking, two KVM virtual machines and couple of docker containers. Everything is working perfectly but after couple of days system is not reachable any more. I found PC running but now keyboard and mouse is working and nothing is displayed on monitor.

There is no useful information in log, I enabled persistent journalctl logs but there is no any information about system crash.

I found only this:

Mar 08 21:35:22 server.com systemd-udevd[217861]: Using default interface naming scheme 'rhel-8.0'.
Mar 08 21:35:22 server.com systemd-udevd[217861]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 08 21:35:22 server.com NetworkManager[825]: <info>  [1678307722.9936] manager: (br-8f108f7485b3): new Bridge device (/org/freedesktop/NetworkManager/Devices/49)
Mar 08 21:35:25 server.com avahi-daemon[810]: Joining mDNS multicast group on interface br-8f108f7485b3.IPv4 with address 172.23.0.1.
Mar 08 21:35:25 server.com kernel: IPv6: ADDRCONF(NETDEV_UP): br-8f108f7485b3: link is not ready
Mar 08 21:35:25 server.com avahi-daemon[810]: New relevant interface br-8f108f7485b3.IPv4 for mDNS.
Mar 08 21:35:25 server.com avahi-daemon[810]: Registering new address record for 172.23.0.1 on br-8f108f7485b3.IPv4.
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7206] device (br-8f108f7485b3): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external>
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7209] device (br-8f108f7485b3): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'exter>
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7214] device (br-8f108f7485b3): Activation: starting connection 'br-8f108f7485b3' (4e5179bf-c077-4937-88e3-68352806c9e0)
Mar 08 21:35:25 server.com dbus-daemon[824]: [system] Activating via systemd: service name='org.freedesktop.nm_dispatcher' unit='dbus-org.freedesktop.nm-dispatcher.service' requested by ':1.7' (u>
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7221] device (br-8f108f7485b3): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7223] device (br-8f108f7485b3): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7224] device (br-8f108f7485b3): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Mar 08 21:35:25 server.com NetworkManager[825]: <info>  [1678307725.7225] device (br-8f108f7485b3): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Mar 08 21:35:25 server.com systemd[1]: Starting Network Manager Script Dispatcher Service...
Mar 08 21:35:26 server.com dbus-daemon[824]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Mar 08 21:35:26 server.com systemd[1]: Started Network Manager Script Dispatcher Service.
Mar 08 21:35:26 server.com NetworkManager[825]: <info>  [1678307726.3550] device (br-8f108f7485b3): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Mar 08 21:35:26 server.com NetworkManager[825]: <info>  [1678307726.3552] device (br-8f108f7485b3): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Mar 08 21:35:26 server.com NetworkManager[825]: <info>  [1678307726.3555] device (br-8f108f7485b3): Activation: successful, device activated.
Mar 08 21:35:26 server.com systemd[1]: iscsi.service: Unit cannot be reloaded because it is inactive.
Mar 08 21:35:27 server.com kernel: br-8f108f7485b3: port 1(vetha34ad22) entered blocking state
Mar 08 21:35:27 server.com kernel: br-8f108f7485b3: port 1(vetha34ad22) entered disabled state
Mar 08 21:35:27 server.com kernel: device vetha34ad22 entered promiscuous mode
Mar 08 21:35:27 server.com kernel: IPv6: ADDRCONF(NETDEV_UP): vetha34ad22: link is not ready
Mar 08 21:35:27 server.com kernel: br-8f108f7485b3: port 1(vetha34ad22) entered blocking state
Mar 08 21:35:27 server.com kernel: br-8f108f7485b3: port 1(vetha34ad22) entered forwarding state
Mar 08 21:35:27 server.com kernel: br-8f108f7485b3: port 1(vetha34ad22) entered disabled state
Mar 08 21:35:27 server.com NetworkManager[825]: <info>  [1678307727.4997] manager: (veth530aac8): new Veth device (/org/freedesktop/NetworkManager/Devices/50)
Mar 08 21:35:27 server.com NetworkManager[825]: <info>  [1678307727.5006] manager: (vetha34ad22): new Veth device (/org/freedesktop/NetworkManager/Devices/51)
Mar 08 21:35:27 server.com systemd-udevd[217913]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 08 21:35:27 server.com systemd-udevd[217913]: Could not generate persistent MAC address for veth530aac8: No such file or directory
Mar 08 21:35:27 server.com systemd-udevd[217915]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable.
Mar 08 21:35:27 server.com systemd-udevd[217915]: Could not generate persistent MAC address for vetha34ad22: No such file or directory

Everything worked fine on Centos 7, PC is quite old with Intel(R) Core(TM) i5-3570K CPU @ 3.40GHz and 32 GB DDR3.

I found in some thread on Internet that people experienced system crash with docker installation.

docker info: 
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.16.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 7
  Running: 6
  Paused: 0
  Stopped: 1
 Images: 8
 Server Version: 23.0.1
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2456e983eb9e37e47538f59ea18f2043c9a73640
 runc version: v1.1.4-0-g5fd4c4d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.18.0-425.13.1.el8_7.x86_64
 Operating System: Rocky Linux 8.7 (Green Obsidian)
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 30.91GiB
 Name: testvm.abcinfo.loc
 ID: 6b35ed44-e416-4b9a-b8d0-38deb0be2ed6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Please help!

Kind regards, Ivan

I tried system reboot several times but problem persists.

I tried searching for some useful logs but nothing useful, I found only that system halted and after that time there is no more logs.

I tried configuring network bridge via NetworkManager and via network-scripts disabling NetworkManager but problem is same.

UPDATE: I enabled kdump and continued to monitor logs, one of KVM virtual machines crashed and there are new errors:

Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: Hardware event. This is not a software error.
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: MCE 0
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: CPU 1 BANK 0 TSC 3ff60bfa4b426
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: TIME 1678763292 Tue Mar 14 04:08:12 2023
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: MCG status:
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: MCi status:
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: Error overflow
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: Corrected error
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: Error enabled
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: MCA: Internal parity error
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: STATUS d000020000010005 MCGSTATUS 0
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: MCGCAP c09 APICID 2 SOCKETID 0
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: MICROCODE 21
Mar 14 04:08:12 testvm.abcinfo.loc mcelog[834]: CPUID Vendor Intel Family 6 Model 58 Step 9
Solution

  • I found solution to problem. After analyzing additional error MCA: Internal parity error, I did RAM check and there was no errors.

    I read that MCA: Internal parity error can be caused by non stable overclock. After checking BIOS configuration there where tree modes of operation quiet, normal and performance. The performance option was selected. After selecting normal mode everything is working as expected.

    It remains to be determined why the performance mode on the Asus board works properly on centos 7 but not on rocky linux 8.7