Search code examples
cwindowsopenssl

When using staticly linked OpenSSL 3.0.8 on some machine legacy provider can't be loaded


Background

I have large C++ project which was using OpenSsl 1.1.1t and now I'm tweaking it to use OpenSsl 3.0.8.

To resolve some problem with loading old pfx files I need legacy provider loaded.

Problem

Following code succeeds on my local machine (Win10):

OpenSslProviderHandler::OpenSslProviderHandler(IReporting* reporter, OSSL_LIB_CTX* ctx)
{
    OPENSSL_init_crypto(OPENSSL_INIT_LOAD_CONFIG, nullptr);

    m_defaultProvider = OSSL_PROVIDER_load(ctx, "default");
    if (m_defaultProvider == nullptr)
    {
        reporter->reportCritical(makeSslError(), "Failed to load the default provider");
    }

    m_legacyProvider = OSSL_PROVIDER_load(ctx, "legacy");
    if (m_legacyProvider == nullptr)
    {
        reporter->reportCritical(makeSslError(), "Failed to load the legacy provider");
    }
}

Nothing is logged.

On testing machines (Win10, Jenkins), same tests are failing. In logs I can see this critical error report:

Failed to load the legacy provider. could not load the shared library (DSO support routines) [asio.ssl:310378599]

Then trying to load private key from pfx file fails: "unsupported (digital envelope routines) [asio.ssl:50856204]".

On my local machine I've checked if some new dll-s are loaded by code shown above, but list of loaded dll-s is same before and after this code is executed.

Tech notes

OpenSsl is build by this script:

perl Configure debug-VC-WIN64A no-asm enable-static-engine no-shared no-tests no-nod-module --prefix="%cd%\%INSTALL_PREFIX%" --openssldir="%cd%\x64\Debug" -FS || exit /b 1

perl -i.bak -pe "s/^\s*@\s*$/    @\$(ECHO\)\n/g" makefile || exit /b 1

nmake -k all  || exit /b 1

nmake install_sw install_ssldirs || exit /b 1

Here is list of dll-s loaded by my program on my local machine when it works:
Sorry this list was incorrect I've misused "Resource monitor" to get this list of library loaded by a process. When I noticed that I've found legacy.dll loaded at <OpenSsl build prefix>\lib\ossl-modules\legacy.dll

ADVAPI32.dll    10.0.19041.3693 C:\WINDOWS\System32\ADVAPI32.dll
bcrypt.dll  10.0.19041.3636 C:\WINDOWS\System32\bcrypt.dll
combase.dll 10.0.19041.3636 C:\WINDOWS\System32\combase.dll
CRYPT32.dll 10.0.19041.3636 C:\WINDOWS\System32\CRYPT32.dll
CRYPTBASE.DLL   10.0.19041.3636 C:\WINDOWS\SYSTEM32\CRYPTBASE.DLL
dbgeng.dll  10.0.19041.3636 C:\WINDOWS\SYSTEM32\dbgeng.dll
dbghelp.dll 10.0.19041.3636 C:\WINDOWS\SYSTEM32\dbghelp.dll
dbgmodel.dll    10.0.19041.3636 C:\WINDOWS\SYSTEM32\dbgmodel.dll
DPAPI.DLL   10.0.19041.3636 C:\WINDOWS\SYSTEM32\DPAPI.DLL
GDI32.dll   10.0.19041.3636 C:\WINDOWS\System32\GDI32.dll
gdi32full.dll   10.0.19041.3636 C:\WINDOWS\System32\gdi32full.dll
IMM32.DLL   10.0.19041.3636 C:\WINDOWS\System32\IMM32.DLL
KERNEL32.DLL    10.0.19041.3636 C:\WINDOWS\System32\KERNEL32.DLL
KERNELBASE.dll  10.0.19041.3636 C:\WINDOWS\System32\KERNELBASE.dll
MPR.dll 10.0.19041.3636 C:\WINDOWS\SYSTEM32\MPR.dll
msi.dll 5.0.19041.3636  C:\WINDOWS\SYSTEM32\msi.dll
msvcp_win.dll   10.0.19041.3636 C:\WINDOWS\System32\msvcp_win.dll
msvcrt.dll  7.0.19041.3636  C:\WINDOWS\System32\msvcrt.dll
MSWSOCK.dll 10.0.19041.3636 C:\WINDOWS\SYSTEM32\MSWSOCK.dll
NETAPI32.dll    10.0.19041.3636 C:\WINDOWS\SYSTEM32\NETAPI32.dll
ntdll.dll   10.0.19041.3636 C:\WINDOWS\SYSTEM32\ntdll.dll
ODBC32.dll  10.0.19041.3636 C:\WINDOWS\SYSTEM32\ODBC32.dll
ole32.dll   10.0.19041.3636 C:\WINDOWS\System32\ole32.dll
OLEAUT32.dll    10.0.19041.3636 C:\WINDOWS\System32\OLEAUT32.dll
RPCRT4.dll  10.0.19041.3636 C:\WINDOWS\System32\RPCRT4.dll
sechost.dll 10.0.19041.3636 C:\WINDOWS\System32\sechost.dll
SHELL32.dll 10.0.19041.3636 C:\WINDOWS\System32\SHELL32.dll
SRVCLI.DLL  10.0.19041.3636 C:\WINDOWS\SYSTEM32\SRVCLI.DLL
ucrtbase.dll    10.0.19041.3636 C:\WINDOWS\System32\ucrtbase.dll
USER32.dll  10.0.19041.3636 C:\WINDOWS\System32\USER32.dll
USERENV.dll 10.0.19041.3636 C:\WINDOWS\SYSTEM32\USERENV.dll
VERSION.dll 10.0.19041.3636 C:\WINDOWS\SYSTEM32\VERSION.dll
win32u.dll  10.0.19041.3636 C:\WINDOWS\System32\win32u.dll
WLDAP32.dll 10.0.19041.3636 C:\WINDOWS\System32\WLDAP32.dll
WS2_32.dll  10.0.19041.3636 C:\WINDOWS\System32\WS2_32.dll
XmlLite.dll 10.0.19041.3636 C:\WINDOWS\SYSTEM32\XmlLite.dll

Questions

  • What can be source of this error?
  • What can be a Win10 difference on my local machine and testing machine (Jenkins), which leads to different behavior (success or failure of loading legacy provider)? Test machine is small virtual Win10 machine with minimum software required.
  • is there some actual dll involved? Which one?
  • or is there some other root cause why legacy provider doesn't load on other system?

Solution

  • Root cause

    Ok I've found the issue. Problem was shared library which was hiding in place where OpenSsl was build. It was exactly at path <build prefix>\lib\ossl-modules\legacy.dll.

    On my local machine it was working since test were run at same machine on which product was build.

    On Jenkins it was failing, since test were run on test machine (multiple configurations) and my product was copied from building machine (docker) to test machine (virtual machines with required software). Copied content didn't cover OpenSsl build location.

    The fix

    Now for a long time I was looking how to make legacy provider part of OpenSsl library and turns out you have to just add one parameter to Configure step: no-module and whole problem has been solved:

    perl Configure debug-VC-WIN64A no-asm enable-static-engine no-shared no-tests no-nod-module no-module --prefix="%cd%\%INSTALL_PREFIX%" --openssldir="%cd%\x64\Debug" -FS || exit /b 1
    

    Disclaimer: no-nod-module parameter is just result of some kind OpenSsl fork in my company (detail is not important).