M2Crypto and java.security return different signed messages

Question

I'm having difficulty porting over some Java code to Python for a project.

The following Java code is used to sign a pseudo-random nonce that's posted to a server for verification:

private static final String NONCE_SIGNATURE_ALGORITHM = "SHA512withRSA";

@JsonProperty
private String encodedData;

@JsonProperty
private String encodedSignedData;

@SuppressWarnings("unused")
private cacertnonceverification() {
}

public cacertnonceverification(byte[] decodedData, PrivateKey privateKey)
        throws NoSuchAlgorithmException, InvalidKeyException, SignatureException {
    byte[] decodedSignedData = sign(privateKey, decodedData);
    encodedData = Base64.getEncoder().encodeToString(decodedData);
    encodedSignedData = Base64.getEncoder().encodeToString(decodedSignedData);
}

@Override
public String toString() {
    final StringBuilder sb = new StringBuilder("CACertNonceVerificationData{");
    sb.append("encodedData='").append(encodedData).append('\'');
    sb.append(", encodedSignedData='").append(encodedSignedData).append('\'');
    sb.append('}');
    return sb.toString();
}

private byte[] sign(PrivateKey privateKey, byte[] dataBytes)
        throws NoSuchAlgorithmException, InvalidKeyException, SignatureException {
    Signature sig = Signature.getInstance(NONCE_SIGNATURE_ALGORITHM);

    sig.initSign(privateKey);
    sig.update(dataBytes);

    return sig.sign();
}

This is working as expected, but the following Python code returns a different value for the same nonce signed by the same private key. I've tested with PyCrypto and M2Crypto - both return the same value when signing a nonce, but they differ from the Java return value.

def sign_nonce(nonce, privKey):

   key=EVP.load_key(privKey)
   key.reset_context(md='sha512')
   key.sign_init()
   key.sign_update(nonce.encode('utf-8'))

   return binascii.b2a_hex(key.sign_final()).decode("utf-8")

def sign_string(message, privKey):

   f = open(privKey, 'r')
   keyString = f.read()
   keyString = keyString.encode('utf-8')
   key = rsa.PrivateKey.load_pkcs1(keyString)

   return binascii.b2a_hex(rsa.sign(message.encode(), key, 'SHA-512')).decode('utf-8')

Any idea what's causing this difference, something I'm missing?

Thanks!

---

edit: My Python script does verify that the signed nonce is valid, and I'm using the same public and private keys for both the Python and Java code, but obviously getting different results for the signed value. That implies to me that the signing algorithm used by java.security and M2Crypto/RSA is different, but I could be way off base - I'm new to crypto.

---

edit 2: I tried using M2Crypto.RSA to sign the nonce, and was getting a still different response, but have modified the code and now M2Crypto EVP and RSA, and rsa are outputting the same signed value. Still no luck getting them to match the expected value from Java:

Expected value: 6d90537599b912ac0de9f68dc0d104a04c5aa74e35262f73872bb01ed1e2cc37100ff1a00735da307fa325f3e30f3c81157755fc60c7ee69bd91bf44b6a5f5e4161bdd67cf982550d992a0a16d12ed5ea30b878a8b2f8eee61cf64e083d7b74a635dabba44bf0b51b9b305aa901b3d090712ee3994057b076d53366a3a387de0f7b2e220cf11239facfce0df6d9a54dfb365f7d3996ea20b55ec75e3661cf95392eda75696cfc6bebda304a2671428dad38a43d5a579550595fe9180539b2b7eb3d568a1f8ce05ded2ae3241a732fffd1404f989eea3e68dda6cd13bfcd6b2e47e64685763a6c3c0f1c47f84c908e22e718254b9f46a7a1de7e2280ccb59c1c5

M2Crypto RSA: 2c973dd7006c058293787504203e36e921ae286f44bcdbf2bd4b48c8a7d9a73723c058d1bcd15b76f6a7145d55cf6109f4bbe5f3953d953177117ef4733e4d3ac22e8a4d320ae6c3a626ddce2a3d5db6b0f10656dd49179c3d60ba912bf21b2e2eac5def74aa6809b74d7711cbb86d719b1fe4df5b7215067af34de3e03859e2c919587bc7f00ec96c6b678c5fc8e114cebae3f5d301811dacc247cd2df9dadf62bccd524147f582f77f0ce29077c7352c8b9893056dce3d65de44d0fc45b88363db9df59c142a01d8f32bf48f05183757e3cd0954c528a7601f4edb56ed9175b219e08aa79514694aedff45bbb365e7d54e534973f6a2aa9f22da12890bbbb5

M2Crypto EVP: 2c973dd7006c058293787504203e36e921ae286f44bcdbf2bd4b48c8a7d9a73723c058d1bcd15b76f6a7145d55cf6109f4bbe5f3953d953177117ef4733e4d3ac22e8a4d320ae6c3a626ddce2a3d5db6b0f10656dd49179c3d60ba912bf21b2e2eac5def74aa6809b74d7711cbb86d719b1fe4df5b7215067af34de3e03859e2c919587bc7f00ec96c6b678c5fc8e114cebae3f5d301811dacc247cd2df9dadf62bccd524147f582f77f0ce29077c7352c8b9893056dce3d65de44d0fc45b88363db9df59c142a01d8f32bf48f05183757e3cd0954c528a7601f4edb56ed9175b219e08aa79514694aedff45bbb365e7d54e534973f6a2aa9f22da12890bbbb5

rsa: 2c973dd7006c058293787504203e36e921ae286f44bcdbf2bd4b48c8a7d9a73723c058d1bcd15b76f6a7145d55cf6109f4bbe5f3953d953177117ef4733e4d3ac22e8a4d320ae6c3a626ddce2a3d5db6b0f10656dd49179c3d60ba912bf21b2e2eac5def74aa6809b74d7711cbb86d719b1fe4df5b7215067af34de3e03859e2c919587bc7f00ec96c6b678c5fc8e114cebae3f5d301811dacc247cd2df9dadf62bccd524147f582f77f0ce29077c7352c8b9893056dce3d65de44d0fc45b88363db9df59c142a01d8f32bf48f05183757e3cd0954c528a7601f4edb56ed9175b219e08aa79514694aedff45bbb365e7d54e534973f6a2aa9f22da12890bbbb5

Answer 1

I was able to resolve the issue by using the EVP and RSA modules together to sign my nonce:

key = RSA.load_key(privKey)
md = EVP.MessageDigest('sha512')
md.update(message)
digest=md.final()
signature = key.sign(digest, "sha512")

return b64encode(signature).decode('utf-8')