How to make JAVA verify OpenSSL RSA/PSS signature

Question

I am working on C RPC client. I need to use OpenSSL for signing mime data, and send the public key and signature to server. Java will try to verify the signature using public key.

Problem is, JAVA failed to verify OpenSSL generated signature but could verify JAVA produced signature. Any idea of how to fix my c client code?

c client code:

#define LENGTH 256
FILE *fp=fopen("ssl.key");
unsigned int uDigestLen = 32;
EVP_PKEY *privkey;
privkey = EVP_PKEY_new();
PEM_read_PrivateKey( fp, &privkey, NULL, NULL);

pRsaKey = EVP_PKEY_get1_RSA(privkey);

if(RSA_check_key(pRsaKey)) {
    printf("RSA key is valid.\n");
}

EVP_MD_CTX_init(&md_ctx);
EVP_DigestInit(&md_ctx, EVP_sha256());
EVP_DigestUpdate(&md_ctx, (const void*) szMessage, strlen(szMessage));
EVP_DigestFinal(&md_ctx, pDigest, &uDigestLen);
EVP_MD_CTX_cleanup(&md_ctx);

// also tried RSA_padding_add_PKCS1_PSS but no luck
status = RSA_padding_add_PKCS1_PSS_mgf1(pRsaKey, EM, pDigest, EVP_sha256(), EVP_sha256(), 20 /* fixed salt length! (tried -2 w/o success)*/);
if (!status)
{
    printf("RSA_padding_add_PKCS1_PSS failed with error %s\n", ERR_error_string(ERR_get_error(), NULL));
}

status = RSA_private_encrypt(LENGTH, EM, pSignature, pRsaKey, RSA_PKCS1_PADDING);
if (status == -1)
{
    printf("RSA_private_encrypt failed with error %s\n", ERR_error_string(ERR_get_error(), NULL));
}

// I send base64 encoded string as well as public key to server, java failed to verify.
cout << "hashedChars: " << base64_encode(pSignature, LENGTH);  

JAVA code:

Signature publicSignature = Signature.getInstance("SHA256withRSA/PSS", "BC");
publicSignature.initVerify(publicKey);
publicSignature.update(text.getBytes(StandardCharsets.UTF_8));

// always fail!!
byte[] signatureBytes = Base64.getDecoder().decode("base64 encoded  from c client");
System.out.println("verified result:" + publicSignature.verify(signatureBytes));

Answer 1

I guess there are two things to fix here:

1. set MGF1 to use SHA-256 explicitly;
2. set the salt size, where OpenSSL uses the largest salt size possible.

See for more info the old discussion here.