Why does OpenSSL bio_do_connect() return 1 even though the connection cannot be established?

Question

we have a problem using OpenSSL 1.1.1n (on windows) related to bio_do_connect(). We created a C# wrapper to OpenSSL so that we can establish an SSL connection with a specific host. If the host is reachable, everything is fine. However, if the host is not reachable, bio_do_connect() blocks for a few seconds and then returns 1 (indicating success) as does bio_do_handshake(). However, the client is definitely not connected, and subsequent reads and writes will always return -1.

We tried placing ERR_print_errors() behind BIO_do_connect() but no errors are printed. I monitored the TCP packages with Wireshark and our machine does never get a response (as expected).

Here is a minimal example of what we are doing:

#include <iostream>
#include "openssl/ssl.h"

const char* remoteEndpoint = "192.168.70.4:25000";

int verify_server_cert(int preverify_ok, X509_STORE_CTX* x509_ctx)
{
  // we always accept the remote certificate for now
  return 1;
}

int main()
{
  const SSL_METHOD* method = TLS_method();
  auto sslContext = SSL_CTX_new(method);
  SSL_CTX_set_verify(sslContext, SSL_VERIFY_PEER, &verify_server_cert);
  SSL_CTX_set_verify_depth(sslContext, 4);
  SSL_CTX_load_verify_locations(sslContext, NULL, "C:\\Users\\Fabian\\source\\P126_Data\\DSMD");

  auto tcpConnection = BIO_new_ssl_connect(sslContext);
  BIO_set_conn_hostname(tcpConnection, remoteEndpoint);
  
  SSL* ssl = nullptr;
  BIO_get_ssl(tcpConnection, &ssl);
  SSL_set_tlsext_host_name(ssl, remoteEndpoint);

  auto returnValue = BIO_do_connect(tcpConnection);
  if (returnValue == 1)
    std::cout << "Connected to " << remoteEndpoint << std::endl;
  else
    std::cout << "Cannot connect (retval = " << returnValue << ")" << std::endl;
}

Notably, if we run this code as a seperate executable, BIO_do_connect() indeed does return a negative value. If we call the exact same code from our DotNet application, it returns 1 however. Is there any explanation for this behaviour?

Thanks in advance!

Answer 1

The error was indeed not in BIO_do_connect() but was caused by the way we used P/Invoke. The return type of our function was bool which apparently is non-blittable. Therefore (BIO_do_connect(...) == 1) evaluated to false but the return value of the DotNet stub was true.