SSL_set_tlsext_host_name crashes

Question

I'm trying to use scio_beast in a project. I understand its rather unfinished, but that should not matter much. I've managed to get it working pretty well.

I'm trying to connect to a server behind CloudFlare now, an I understand I need SNI for that to work.

Given the following:

using tcp           = boost::asio::ip::tcp;     //  <boost/asio/ip/tcp.hpp>
namespace websocket = boost::beast::websocket;  //  <boost/beast/websocket.hpp>
namespace ssl       = boost::asio::ssl;         //  <boost/asio/ssl.hpp>

typedef websocket::stream<ssl::stream<tcp::socket>>  SecureWebSocket;
typedef std::unique_ptr<SecureWebSocket>            SecureWebSocketPtr;

// ...

SecureWebSocketPtr m_wss;

m_sslContext = std::make_shared<ssl::context>(ssl::context::tlsv12_client);
m_wss.reset(new SecureWebSocket(m_ios, *m_sslContext.get()));

auto ssl = m_wss->next_layer().native_handle();
auto host = m_connectOptions.host.c_str();
SSL_set_tlsext_host_name(m_wss->next_layer().native_handle(), host);

It does not seem to matter much where I call SSL_set_tlsext_host_name (e.g. before resolving, before async_handshake), it always crashes trying to execute 0x0000000000000000.

Answer 1

Turns out this was a linking problem: The app uses another library (WebRTC) that uses boringssl. Somehow the linker didn't complain about duplicate symbols when linking OpenSSL after WebRTC and silently used boringssl's functions. Both MSVC and gcc do it.

I solved it by moving the websocket/OpenSSL code into a dll, which allows it to properly link against OpenSSL independently from the app.

Not the prettiest solution, but building WebRTC with OpenSSL doesn't seem to be really supported or at least maintained.