How to compute a shared secret for 2 users in algorithm Diffie Hellman using openssl lib c language?

Question

I need some help with algorithm Diffie Hellman in openssl I have prime number (p), generator (g), private key of user A and public key of user B. I need to compute the shared key. I wrote this code, but the code is executed till this line

 int dhSize = DH_size(dh->priv_key);

Here is the full code:

#include <stdio.h>
#include <openssl/dh.h>

const char* userA_PrivateKey = "90ff0";
const char* userB_PublicKey = "9d1a59";
const char* p = "66c2fa";
const char* g = "2";

int main(void)
{
    DH *dh = DH_new();

    BN_dec2bn(&dh->g, g);
    BN_hex2bn(&dh->p, p);
    BN_hex2bn(&dh->priv_key, userA_PrivateKey);

    BIGNUM *pubKeyUserB = NULL;
    BN_dec2bn(&pubKeyUserB, userB_PublicKey);

    //Compute the shared secret
    int secret_size;
    unsigned char *secret;
    printf(" Compute DH_size \n");
    int dhSize = DH_size(dh->priv_key);
    printf(" dhSize = %d \n"); //NOT EXECUTED 
    secret = OPENSSL_malloc(sizeof(unsigned char) * dhSize);

    if(0 > (secret_size = DH_compute_key(secret, pubKeyUserB, dh->priv_key)))
    {
        printf("error \n");
    }

    return 0;
}

I have two problems:

1) printf, that prints dhSize is not executed at all

2) I am not sure if I set values g, p, priv key correctly ? Will the function DH_compute_key use my g and p ?

Answer 1

You are making silly mistakes:

1. dhSize should be typed as DH_size (~line 24) and DH_size function calculates size of struct DH given a const struct DH * you are passing it dh->priv_key instead of passing it dh (~line 28)

2. Similar mistake in use of DH_compute_key (~line 28) third argument should be dh not dh->priv_key.

Please fix accordingly and try again