Verify digital signature using BIGNUM library

Question

why is my code returning weird result?

void printBN(char *msg, BIGNUM * a)
{
    char * number_str = BN_bn2hex(a);
    printf("%s %s\n", msg, number_str);
    OPENSSL_free(number_str);
}

int main () 
{
    BN_CTX *ctx = BN_CTX_new();

    BIGNUM *e = BN_new();
    BIGNUM *n = BN_new();
    BIGNUM *d = BN_new();
    BIGNUM *plaintxt = BN_new();
    BIGNUM *signature = BN_new();

    BN_hex2bn(&signature, "239a09ea0d5fdaea94ec97130b1c74c89764226065bbe614da7fb9f851be7beabd5f8ece4b06c84c8bf49162e2c914fc8c8b6bd98b9ed086fb3bdb7f74fc5075424541fc8911c6c87953d5377d38fff94ecb0cb86018d3f00c5a1d418cf9e65e19dd56a947ae301fc73ffc9d513ece60e85373a4408f6958a466e41044a3b924d3e85c4ea6afaa86b2cfe388f70873d8513c1c10df65ccdfdb00cbd7d7a5f08dc7f3e5f40b500ac7913796dd11a71d88aaf53f728ab1584b80ea7e32e378679b247297df7b562b08efd4a56e1861f41dcd1d17d58e1015a8c5c580862167e884876ce72ff539e8da7397bd86cca42ac02576b778ad4a96943c6ae1b33f27a8dd");
    BN_hex2bn(&n, "D018CF45D48BCDD39CE440EF7EB4DD69211BC9CF3C8E4C75B90F3119843D9E3C29EF500D10936F0580809F2AA0BD124B02E13D9F581624FE309F0B747755931D4BF74DE1928210F651AC0CC3B222940F346B981049E70B9D8339DD20C61C2DEFD1186165E7238320A82312FFD2247FD42FE7446A5B4DD75066B0AF9E426305FBE01CC46361AF9F6A33FF6297BD48D9D37C1467DC75DC2E69E8F86D7869D0B71005B8F131C23B24FD1A3374F823E0EC6B198A16C6E3CDA4CD0BDBB3A4596038883BAD1DB9C68CA7531BFCBCD9A4ABBCDD3C61D7931598EE81BD8FE264472040064ED7AC97E8B9C05912A1492523E4ED70342CA5B4637CF9A33D83D1CD6D24AC07");
    BN_hex2bn(&e, "10001");

    BN_mod_exp(plaintxt, signature, e, n, ctx);
    printBN("hash value: \n", plaintxt);

}

My code gives the following result, which doesn't look like correct result of the calculation:

01FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF003031300D060960864801650304020105000420BC5F9353CBB9DCAE86B9F8F68C1C95856DB836ACA2E00C9319716CDF4DD0F5BA

Answer 1

Looks like the correct result to me, except that you are missing the initial byte, because it is defined (PKCS#1 v2.2) to be 00:

Header byte:

01

PKCS#1 v1.5 padding characters:

FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF

end of padding character:

00

ASN.1 encoding of hash OID and hash value:

3031300D060960864801650304020105000420BC5F9353CBB9DCAE86B9F8F68C1C95856DB836ACA2E00C9319716CDF4DD0F5

which decodes (Lapo's ASN.1 decoder) to:

SEQUENCE (2 elem)
  SEQUENCE (2 elem)
    OBJECT IDENTIFIER 2.16.840.1.101.3.4.2.1 sha-256 (NIST Algorithm)
    NULL
  OCTET STRING (32 byte) BC5F9353CBB9DCAE86B9F8F68C1C95856DB836ACA2E00C9319716CDF4DD0F5BA

Where the contents of the octet string (byte array in hex) is the hash value.

---

I guess it is just a bit more complex than you initially thought. Try PSS, it is even more complex (but also more secure). Note that there are many attacks on "raw" or "textbook" RSA (just modular exponentiation). Such padding is required to have a secure signature or - in case of encryption - ciphertext.