Search code examples
linuxcentosyumcentos8

Illegal instruction (core dumped) after packages upgrade on Centos 8


Some problems with my centos 8 began when i entered "yum update -y" command. After that i started receive Illegal instruction (core dumped) messages when i run yum or any python scripts.

During upgrade i got such errors:

  Upgrading        : NetworkManager-libnm-1:1.30.0-0.5.el8.x86_64                                                        18/168 
  Running scriptlet: NetworkManager-libnm-1:1.30.0-0.5.el8.x86_64                                                        18/168 
  Running scriptlet: NetworkManager-1:1.30.0-0.5.el8.x86_64                                                              19/168 
  Upgrading        : NetworkManager-1:1.30.0-0.5.el8.x86_64                                                              19/168 
  Running scriptlet: NetworkManager-1:1.30.0-0.5.el8.x86_64                                                              19/168 
/var/tmp/rpm-tmp.1WBtNY: line 4:  4581 Illegal instruction     (core dumped) firewall-cmd --reload --quiet

  Upgrading        : elfutils-libs-0.182-3.el8.x86_64                                                                    20/168 
  Upgrading        : elfutils-debuginfod-client-0.182-3.el8.x86_64                                                       21/168 
  Upgrading        : python3-syspurpose-1.28.9-1.el8.x86_64                                                              22/168 

  Running scriptlet: selinux-policy-targeted-3.14.3-59.el8.noarch                                                        58/168 
  Upgrading        : selinux-policy-targeted-3.14.3-59.el8.noarch                                                        58/168 
  Running scriptlet: selinux-policy-targeted-3.14.3-59.el8.noarch                                                        58/168 
  Upgrading        : subscription-manager-1.28.9-1.el8.x86_64                                                            59/168 
  Running scriptlet: subscription-manager-1.28.9-1.el8.x86_64                                                            59/168 
chmod: cannot access '/etc/pki/entitlement/*.pem': No such file or directory

  Upgrading        : lvm2-8:2.03.11-0.4.20201222gitb84a992.el8.x86_64                                                    60/168 
  Running scriptlet: lvm2-8:2.03.11-0.4.20201222gitb84a992.el8.x86_64                                                    60/168 
  Upgrading        : kpartx-0.8.4-7.el8.x86_64                                                                           61/168 
  Upgrading        : xfsprogs-5.0.0-7.el8.x86_64                                                                         62/168 


  Cleanup          : elfutils-default-yama-scope-0.182-2.el8.noarch                                                     154/168 
  Cleanup          : tzdata-java-2020d-1.el8.noarch                                                                     155/168 
  Cleanup          : samba-common-4.13.2-5.el8.noarch                                                                   156/168 
  Running scriptlet: NetworkManager-1:1.30.0-0.4.el8.x86_64                                                             157/168 
  Cleanup          : NetworkManager-1:1.30.0-0.4.el8.x86_64                                                             157/168 
  Running scriptlet: NetworkManager-1:1.30.0-0.4.el8.x86_64                                                             157/168 
/var/tmp/rpm-tmp.M7HXpR: line 4:  9248 Illegal instruction     (core dumped) firewall-cmd --reload --quiet

  Cleanup          : NetworkManager-libnm-1:1.30.0-0.4.el8.x86_64                                                       158/168 
  Running scriptlet: NetworkManager-libnm-1:1.30.0-0.4.el8.x86_64                                                       158/168 
  Cleanup          : perl-libs-4:5.26.3-417.el8.x86_64                                                                  159/168 


  Cleanup          : python3-libs-3.6.8-33.el8.x86_64                                                                   164/168 
  Cleanup          : glibc-2.28-141.el8.x86_64                                                                          165/168 
  Cleanup          : glibc-langpack-en-2.28-141.el8.x86_64                                                              166/168 
  Cleanup          : glibc-common-2.28-141.el8.x86_64                                                                   167/168 
  Cleanup          : tzdata-2020d-1.el8.noarch                                                                          168/168 
  Running scriptlet: libwbclient-4.13.3-1.el8.x86_64                                                                    168/168 
  Running scriptlet: tuned-2.15.0-1.el8.noarch                                                                          168/168 
  Running scriptlet: kmod-kvdo-6.2.4.26-76.el8.x86_64                                                                   168/168 
sort: fflush failed: 'standard output': Broken pipe
sort: write error

gzip: stdout: Broken pipe

gzip: stdout: Broken pipe
sort: write failed: 'standard output': Broken pipe
sort: write error
sort: fflush failed: 'standard output': Broken pipe
sort: write error

gzip: stdout: Broken pipe

gzip: stdout: Broken pipe
sort: write failed: 'standard output': Broken pipe
sort: write error

  Running scriptlet: tzdata-2020d-1.el8.noarch                                                                          168/168 
[/usr/lib/tmpfiles.d/pesign.conf:1] Line references path below legacy directory /var/run/, updating /var/run/pesign вЖТ /run/pesign; please update the tmpfiles.d/ drop-in file accordingly.

  Running scriptlet: glibc-common-2.28-145.el8.x86_64                                                                   168/168 
  Verifying        : python3-pexpect-4.3.1-3.el8.noarch                                                                   1/168 
  Verifying        : python3-ptyprocess-0.5.2-4.el8.noarch                                                                2/168 
  Verifying        : abattis-cantarell-fonts-0.0.25-6.el8.noarch                                                          3/168 


Now problems are with yum or with import of some python packages. But curl is ok.

[root@ddns-ard-2 ~]# 
[root@ddns-ard-2 ~]# yum --version    
Illegal instruction (core dumped)
[root@ddns-ard-2 ~]# yum upgrade   
Illegal instruction (core dumped)
[root@ddns-ard-2 ~]# python3
Python 3.6.8 (default, Dec 22 2020, 19:04:08) 
[GCC 8.4.1 20200928 (Red Hat 8.4.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> import pymongo
Illegal instruction (core dumped)
[root@ddns-ard-2 ~]# rpm -q curl
curl-7.61.1-17.el8.x86_64
[root@ddns-ard-2 ~]# curl https://www.google.com | more
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!doctype html>

Messages from my /var/log/messages

Jan 12 02:18:02 ddns-ard-2 kernel: traps: yum[3731] trap invalid opcode ip:7efdef6dd59f sp:7ffef19166c0 error:0 in libm-2.28.so[7efdef655000+181000]
Jan 12 02:18:02 ddns-ard-2 systemd[1]: Started Process Core Dump (PID 3732/UID 0).
Jan 12 02:18:02 ddns-ard-2 systemd-coredump[3733]: Resource limits disable core dumping for process 3731 (yum).
Jan 12 02:18:02 ddns-ard-2 systemd-coredump[3733]: Process 3731 (yum) of user 0 dumped core.
Jan 12 02:18:02 ddns-ard-2 systemd[1]: systemd-coredump@278-3732-0.service: Succeeded.

Workaround i founded in internet not helped

[root@ddns-ard-2 ~]# export NSS_DISABLE_HW_AES=1 
[root@ddns-ard-2 ~]# yum --version
Illegal instruction (core dumped)
[root@ddns-ard-2 ~]# export NSS_DISABLE_HW_GCM=1
[root@ddns-ard-2 ~]# yum --version              
Illegal instruction (core dumped)
[root@ddns-ard-2 ~]# 
[root@ddns-ard-2 ~]# dnf
Illegal instruction (core dumped)
[root@ddns-ard-2 ~]# 

Any ideas how to resolve this problem ?

I have 5 virtual CPUs with the same cores here is information about one of them:

[root@ddns-ard-2 ~]#  cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.30GHz
stepping        : 2
microcode       : 0x38
cpu MHz         : 2294.686
cache size      : 46080 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 15
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc cpuid aperfmperf pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx hypervisor lahf_lm epb pti cqm_llc cqm_occup_llc dtherm ida arat pln pts
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips        : 4589.37
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

Solution

  • This happens if the hypervisor masks XSAVE support, but does not mask the FMA feature bit in CPUID. It's a bit of a weird hypervisor configuration, which is why it was not caught in testing. Selecting a different CPU model in the hypervisor should work around this (something that supports AVX2 should do it). As a side effect, you will likely get better performance from your VMs.

    Some reported this through appropriate channels, and updates have been released: