After using PEM_read_bio_PUBKEY I am unable to unload win dll

I am working on a Windows DLL which is being loaded by a 3rd party executable. It has a stop functionality which unloads my DLL. I can see that while running in Visual Studio's debugger:

'3rdparty.exe' (Win32): Unloaded 'C:\my_dll.dll'

Now, if I add this code:

const std::string& public_key = my_public_key;
auto bio = BIO_new(BIO_s_mem());

if (!BIO_write(bio, public_key.c_str(), static_cast<int>(public_key.size())))
{
  error();
}

EVP_PKEY* pkey = nullptr; 
if (!PEM_read_bio_PUBKEY(bio, &pkey, nullptr, nullptr) || pkey == nullptr)
{
  error();
}

if (pkey) 
{
  EVP_PKEY_free(pkey);
}

if (bio)
  BIO_free_all(bio);

Stopping my DLL, unloading doesn't happen anymore.

I tried a few things, like using a custom context:

  OSSL_LIB_CTX* lib_ctx = OSSL_LIB_CTX_new();
  if (!PEM_read_bio_PUBKEY_ex(bio, &pkey, nullptr, nullptr, lib_ctx, nullptr) || pkey == nullptr)
  {
    error();
  }
  ...
  OSSL_LIB_CTX_free(lib_ctx);

Also tried various clean up methods:

EVP_cleanup();                // Clean up EVP layer (legacy compatibility)
CRYPTO_cleanup_all_ex_data(); // Clean up extra data
ERR_free_strings();           // Free error strings
SSL_COMP_free_compression_methods(); // Free compression methods (if SSL is used)

But nothing helps.

I do not know exactly how the 3rd party does the stopping of my DLL, and I do not have access to that code.

SSL that I am using is: OpenSSL 3.2.1 30 Jan 2024

I statically link against libcrypto.lib and libssl.lib libraries.

Solution

By default, OpenSSL "pins" itself in memory so it can't be unloaded until process termination. However, until recently, this behavior was unconditional regardless of whether you used the static or dynamic version of OpenSSL. Which means, if you static link OpenSSL into your own DLL, then your DLL becomes pinned - just like you are experiencing.

There are some (closed) tickets on OpenSSL's GitHub repo about this issue:

#7598 dll can't be free after it call bio_new and bio_free APIs in windows platform

#20977 Static libcrypto uses DLL pinning

To address the issue, you need to:

use a version of the OpenSSL static libs which have been compiled with the no-shared and no-pinshared options.
have your code call OPENSSL_init_crypto() with the OPENSSL_INIT_NO_ATEXIT option (you will then have to call OPENSSL_cleanup() manually).

This is documented in OpenSSL's install notes:

Enable and Disable Features

Feature options always come in pairs, an option to enable feature xxxx, and an option to disable it:

[ enable-xxxx | no-xxxx ]

Whether a feature is enabled or disabled by default, depends on the feature. In the following list, always the non-default variant is documented: if feature xxxx is disabled by default then enable-xxxx is documented and if feature xxxx is enabled by default then no-xxxx is documented.

...

no-pinshared

Don't pin the shared libraries.

By default OpenSSL will attempt to stay in memory until the process exits. This is so that libcrypto and libssl can be properly cleaned up automatically via an atexit() handler. The handler is registered by libcrypto and cleans up both libraries. On some platforms the atexit() handler will run on unload of libcrypto (if it has been dynamically loaded) rather than at process exit.

This option can be used to stop OpenSSL from attempting to stay in memory until the process exits. This could lead to crashes if either libcrypto or libssl have already been unloaded at the point that the atexit handler is invoked, e.g. on a platform which calls atexit() on unload of the library, and libssl is unloaded before libcrypto then a crash is likely to happen.

Note that shared library pinning is not automatically disabled for static builds, i.e., no-shared does not imply no-pinshared. This may come as a surprise when linking libcrypto statically into a shared third-party library, because in this case the shared library will be pinned. To prevent this behaviour, you need to configure the static build using no-shared and no-pinshared together.

Applications can suppress running of the atexit() handler at run time by using the OPENSSL_INIT_NO_ATEXIT option to OPENSSL_init_crypto(). See the man page for it for further details.

...

no-shared

Do not create shared libraries, only static ones.

...

Which means pinshared is enabled by default, even in static builds.