Getting crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE when
dracut-fips package is installed and sysctl -a shows crypto.fips_enabled = 1
Restriction: Can not disable dracut-fips package or crypto.fips_enable setting in my case.
How can I get cmake --version working inside the container?
someLinuxUser@jenkins-project_team_rh ~]$ hostname -f; hostname -i
jenkins-project_team_rh.lewisville.us.company.com
10.20.20.10[someLinuxUser@jenkins-project_team_rh ~]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
[someLinuxUser@jenkins-project_team_rh ~]$
[someLinuxUser@jenkins-project_team_rh ~]$ cmake --version
cmake version 3.18.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
[someLinuxUser@jenkins-project_team_rh ~]$ sudo docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
opensuse-image 15.2 618840498a55 3 hours ago 2.59GB
NOW I'm inside the container. Running "cmake --version" here fails with an error.
NOTE: The same docker image will work fine if you have ANY host OS, where "sysctl -a | grep fips" will show "crypto.fips_enabled = 0" in it's output. So, we may need to have this value set as 0.
[someLinuxUser@jenkins-project_team_rh ~]$ sudo docker run -it opensuse-image:15.2 bash
Active Directory Password:
WARNING: IPv4 forwarding is disabled. Networking will not work.
docker_nonroot_user@eaa40032f4d3:~/git>
docker_nonroot_user@eaa40032f4d3:~/git> which cmake; cmake --version
/usr/bin/cmake
crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Aborted (core dumped)
docker_nonroot_user@eaa40032f4d3:~/git>
docker_nonroot_user@3e63938cf7e7:~/git> cat /etc/os-release
NAME="openSUSE Leap"
VERSION="15.2"
ID="opensuse-leap"
ID_LIKE="suse opensuse"
VERSION_ID="15.2"
PRETTY_NAME="openSUSE Leap 15.2"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:opensuse:leap:15.2"
BUG_REPORT_URL="https://bugs.opensuse.org"
HOME_URL="https://www.opensuse.org/"
[someLinuxUser@jenkins-project_team_rh ~]$ sysctl -a | grep fips_enabled
crypto.fips_enabled = 1
i.e. it shows, fips is enabled on this HOST machine. I think if this (crypto.fips_enabled was set to = 0) then we may not see this issue with cmake, but in my case, I can't disable this setting.
On the Host machine, some related packages are:
[someLinuxUser@jenkins-project_team_rh ~]$ sudo yum list installed | egrep "fips|openssl|dracut"
dracut-fips.x86_64 033-572.el7 @rhel-x86_64-server-7
fipscheck.x86_64 1.4.1-6.el7 @anaconda/7.6
fipscheck-lib.x86_64 1.4.1-6.el7 @anaconda/7.6
CentrifyDC-openssl.x86_64 5.7.1-347 installed
openssl.x86_64 1:1.0.2k-22.el7_9 @q1_rhel-x86_64-server-7
openssl-devel.x86_64 1:1.0.2k-22.el7_9 @q1_rhel-x86_64-server-7
openssl-libs.x86_64 1:1.0.2k-22.el7_9 @q1_rhel-x86_64-server-7
openssl098e.x86_64 0.9.8e-29.el7_2.3 @anaconda/7.6
xmlsec1-openssl.x86_64 1.2.20-7.el7_4 @anaconda/7.6
dracut.x86_64 033-572.el7 @rhel-x86_64-server-7
dracut-config-rescue.x86_64 033-572.el7 @rhel-x86_64-server-7
dracut-network.x86_64 033-572.el7 @rhel-x86_64-server-7
[someLinuxUser@jenkins-project_team_rh ~]$
Using same Docker image on a different machine, same HOST OS.
On this another RH 7.9 OS host machine I have, I have the same docker image SCP'ed, when I start the same OpenSuse 15.2 image as docker container and run cmake --version, everything works fine and I don't see this error.
Running:
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ hostname -f; hostname -i
rh_7_9_os_machine.company.local
10.100.100.10
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.9 (Maipo)
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
opensuse-image 15.2 618840498a55 3 hours ago 2.59GB
This HOST shows:
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sysctl -a 2>/dev/null | grep fips_enabled
crypto.fips_enabled = 0
Running Docker image --> container and cmake --version in it, works!
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo docker run -it opensuse-image:15.2 bash
docker_nonroot_user@fb751d198066:~/git>
docker_nonroot_user@fb751d198066:~/git> sysctl -a 2>/dev/null | grep fips
crypto.fips_enabled = 0
docker_nonroot_user@fb751d198066:~/git>
docker_nonroot_user@fb751d198066:~/git> cmake --version
cmake version 3.17.0
CMake suite maintained and supported by Kitware (kitware.com/cmake).
docker_nonroot_user@fb751d198066:~/git> exit
YUM packages on HOST machine are:
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep fips
fipscheck.x86_64 1.4.1-6.el7 @anaconda/7.4
fipscheck-lib.x86_64 1.4.1-6.el7 @anaconda/7.4
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep openssl
openssl.x86_64 1:1.0.2k-22.el7_9 @rhel-7-server-rhui-rpms
openssl-devel.x86_64 1:1.0.2k-22.el7_9 @rhel-7-server-rhui-rpms
openssl-libs.x86_64 1:1.0.2k-22.el7_9 @rhel-7-server-rhui-rpms
openssl11-libs.x86_64 1:1.1.1g-2.el7 @epel
xmlsec1-openssl.x86_64 1.2.20-7.el7_4 @rhui-REGION-rhel-server-releases
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep dracut\-fips
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$
[gigauser@rh_7_9_os_machine opensuse-x-project_team_-mse]$ sudo yum list installed |grep dracut
dracut.x86_64 033-572.el7 @rhel-7-server-rhui-rpms
dracut-config-generic.x86_64 033-572.el7 @rhel-7-server-rhui-rpms
dracut-config-rescue.x86_64 033-572.el7 @rhel-7-server-rhui-rpms
dracut-network.x86_64 033-572.el7 @rhel-7-server-rhui-rpms
As you can see above, there's NO package named dracut-fips installed on this machine, assuming that's why crypto.fips_enabled = 0 and "cmake --version" works on this HOST and within container as well!!
How can I get cmake --version working inside the container? when I can't uninstall dracut-fips package and disable FIPS by having crypto.fips_enabled = 0 on the first HOST OS.
Found the solution
WHY this issue was coming:
Docker build (to create the image) was run on a machine (the other machine, where cmake was working inside the docker container)... i.e. where FIPS was not installed.
During the build step, Docker file was running
zypper install cmake (or yum install cmake).
As FIPS was disabled on the machine where docker image was built, cmake was installed in docker image, without understanding FIPS being enabled / dracut-fips being installed.
Then, when you copy this image and consume it on a machine where FIPS is actually ENABLED, then cmake was failing with that error mesg: crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
--
Solution #1: Tag your docker image appropriately.
Summary:
PS: If you do install cmake using the above, then it'll only work on target machine where you run container, if that machine's FIPS is disabled or enabled at build time. i.e. if FIPS was enabled, you install cmake and run it on a machine where it's NOT same FIPS setting as the host, where image was built, then you'll be reading this post for help.
Better way in case you want to install cmake using the above package managers, would be, tag your docker image appropriately during image creation time i.e.:
docker build -t <image-name>-fips-enabled ... if FIPS is enabled
and
docker build -t <image-name>-fips-disabled ... if FIPS on that that machine is disabled.
That way, you can pick the correct docker image imagename-fips-enabled vs imagename-fips-disabled acc. to what your target machine FIPS setting is (where you'll actually perform docker run ... using this image).
--
Solution #2: Don't use zypper (OpenSuse) or yum if you have RedHat container. and this solution is flexible in the sense, that it's independent of FIPS setting = 0 / 1 on the host, where image was built.
I didn't use zypper/yum to install cmake inside Dockerfile, but just grabbed cmake-3.18.2-Linux-x86_64.tar.gz bundle file.
Within Dockerfile, I simply extracted this .tar.gz file inside some directory.
Also I set export PATH:/path/where/I/installed/cmake-3.18.2../bin:/..some_other_paths:/...:/.... in Dockerfile inside RUN statement.
i.e.
RUN export PATH=/path/where/I/installed/cmake-3.18.2../bin:/...... && <more cmds here> && <some other cmds here> && ... etc, so it can find the extracted cmake 3.18.2 for any buil-time (cmake operations) and also SET the same PATH=/... variable as ENV PATH=/.... same value used during RUN for PATH so at runtime, when the container runs, $PATH is all set for finding cmake (3.18.2 version) rather than using any existing /usr/bin/cmake or some other shit).
Dockerfile snapshot:
# curl -k -sSf -H "X-JFrog-Art-Api:dslfhjlieurqwihlj233lk2l4j6p9usdkajdfasddl809842iijhlkhflhafOHIHFLyeaGoodLuck" \
# -o /tmp/cmake.tar.gz https://artifactory.company.com/artifactory/some-Local/cmake/cmake-3.18.2-Linux-x86_64.tar.gz && \
and
as my umask settings were set to 022, I didn't have to do any chicken chmod operation post untar:
# echo -e "\n-- Installing CMake ...\n" && \
# tar -xvzpf /tmp/cmake.tar.gz -C /home/docker_nonroot_user/tools/ && \
Inside the docker container, as ENV PATH=/... was set in Dockerfile for this target path as well, the correct cmake 3.18.2 was used at docker container runtime action.
cmake installed location in my case was:
/home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake
PATH inside docker container was:
/home/docker_nonroot_user/tools/cov-analysis/bin:/home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Using a Docker image built using SOLUTION #2 above, i.e. docker image built on a machine where FIPS = 0 aka DISABLED and then later, using this same image to create a container on a totally different target host machine where FIPS = 1 aka ENABLED, I see:
87d8104d8c41:/home/docker_nonroot_user # sysctl -a|grep fips_en
crypto.fips_enabled = 1
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # which cmake
/usr/bin/cmake
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # cmake --version
crypto/fips/fips.c:153: OpenSSL internal error: FATAL FIPS SELFTEST FAILURE
Aborted (core dumped)
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # ls -l /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin
total 75504
-rwxr-xr-x 1 root root 11908568 Aug 20 2020 ccmake
-rwxr-xr-x 1 root root 12096216 Aug 20 2020 cmake
-rwxr-xr-x 1 root root 27476480 Aug 20 2020 cmake-gui
-rwxr-xr-x 1 root root 12398808 Aug 20 2020 cpack
-rwxr-xr-x 1 root root 13318712 Aug 20 2020 ctest
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # ls -l /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake
-rwxr-xr-x 1 root root 12096216 Aug 20 2020 /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake
87d8104d8c41:/home/docker_nonroot_user #
87d8104d8c41:/home/docker_nonroot_user # /home/docker_nonroot_user/tools/cmake-3.18.2-Linux-x86_64/bin/cmake --version
cmake version 3.18.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
87d8104d8c41:/home/docker_nonroot_user # : Merry X-mas Baaaaeeebyyy! - no more FIPS shit error now. Next I'll fix some chown on ~<user> and close my story.
SOLUTION #3: You can disable FIPS but only if you are allowed to do so, then you don't need solution #1 or solution #2.
The effects of self-test failures in the Module differ depending on the type of self-test that failed.
The FIPS_mode_set() function verifies the integrity of the runtime executable using a HMAC SHA-256 digest, which is computed at build time. If this computed HMAC SHA-256 digest matches the stored, known digest, then the power-up self-test (consisting of the algorithm-specific Pairwise Consistency and Known Answer tests) is performed.
Non-fatal self-test errors transition the module into an error state. The application must be restarted to recover from these errors. The non-fatal self-test errors are:
FIPS_R_FINGERPRINT_DOES_NOT_MATCH - The integrity verification check failed
FIPS_R_FIPS_SELFTEST_FAILED - a known answer test failed
FIPS_R_SELFTEST_FAILED - a known answer test failed
FIPS_R_TEST_FAILURE – a known answer test failed (RSA); pairwise consistency test failed (DSA)
FIPS_R_PAIRWISE_TEST_FAILED – a pairwise consistency test during DSA or RSA key generation failed
FIPS_R_FIPS_MODE_ALREADY_SET - the application initializes the FIPS mode when it is already initialized
These errors are reported through the regular ERR interface of the shared libraries and can be queried by functions such as ERR_get_error(). See the OpenSSL Module manual page for the function description.
A fatal error occurs only when the module is in the error state (a self-test has failed) and the application calls a crypto function of the module that cannot return an error in normal circumstances (void return functions). The error message: 'FATAL FIPS SELFTEST FAILURE' is printed to stderr and the application is terminated with the abort() call.
The only way to recover from a fatal error is to restart the application. If failures persist, you must reinstall the Module. If you downloaded the software, verify the package hash to confirm a proper download.
If you installed a newer version of OpenSSL x.y.z (rpm/etc) and didn't reboot/restart your machine, some of the libraries which consume openssl will be vulnerable / error out too with similar errors.
You can test this by running:
uptime command to see how long your server/machine has been UP n running.sudo lsof | grep libssl.so or to Find processes running with deleted OpenSSL libraries, run:sudo lsof | grep DEL.*libssl
So, try rebooting your machine and see if the errors goes away and check the output of the above commands.
For more info, refer this link: https://raymii.org/s/snippets/Find_all_services_using_libssl_to_restart_after_an_OpenSSL_update.html