How can I validate hostnames with OpenSSL 1.1.0?

I'm trying to implement a sample hostname validation with OpenSSL. The sample C/C++ code I have put together is:

// please note I'm connecting to https://openssl.org itself
// enable SNI
if(!SSL_set_tlsext_host_name(ssl, "www.openssl.org")) throw;
if(!SSL_connect(ssl)) throw;
// connection is fine, I can get the homepage via HTTP
X509 *cert = SSL_get_peer_certificate(ssl);
if(cert) {
    if(!X509_VERIFY_PARAM_set1_host(SSL_get0_param(ssl), "google.com", 0)) throw;
    SSL_set_verify(ssl, SSL_VERIFY_PEER, 0);
    const long cert_res = SSL_get_verify_result(ssl);
    if(cert_res == X509_V_OK) {
        printf("Certificate verified!\n");
    }
    X509_free(cert);
}

As per code above I'm successfully connecting to the openssl.org domain; then I'm setting the name to verify as google.com to test failures, but the code still succeeds.

What am I doing wrong? How can I implement a thorough verification of hostnames using OpenSSL APIs? I wouldn't want to re-implement (most likely with bugs/wrongly) what is already implemented in the library...

I'm using Ubuntu 16.04 and this libssl version: /lib/x86_64-linux-gnu/libssl.so.1.0.0.

Solution

As suggested by jww, one simply needs to set (at least)

if(!X509_VERIFY_PARAM_set1_host(SSL_get0_param(ssl), "google.com", 0)) throw;

before performing the connection itself:

if(!SSL_connect(ssl)) throw;

In this case, one case ensure OpenSSL to implement the check automatically at connection time by adding following code:

SSL_set_verify(ssl, SSL_VERIFY_PEER, 0);

before calling SSL_connect, or follow the same path as before and have return X509_V_ERR_HOSTNAME_MISMATCH by SSL_get_verify_result if one wants to handle things in more details:

// please note I'm connecting to https://openssl.org itself
// enable SNI
if(!SSL_set_tlsext_host_name(ssl, "www.openssl.org")) throw;
// enable name/domain verification
if(!X509_VERIFY_PARAM_set1_host(SSL_get0_param(ssl), "google.com", 0)) throw;
if(!SSL_connect(ssl)) throw;
// get certificate
X509 *cert = SSL_get_peer_certificate(ssl);
if(cert) {
    const long cert_res = SSL_get_verify_result(ssl);
    // in case of name/domain mismatch cert_res will
    // be set as 62 --> X509_V_ERR_HOSTNAME_MISMATCH
    if(cert_res != X509_V_OK) throw; // certificate has been tampered with
    X509_free(cert);
} else throw; // we couldn't get certificate