Errors using python's ECDSA lib to sign AWS messages

I'm signing my messages using my code below:

def sign_msg_hash(self, msg_hash: HexBytes):
        signature = self._kms_client.sign(
            KeyId=self._key_id,
            Message=msg_hash,
            MessageType="DIGEST",
            SigningAlgorithm="ECDSA_SHA_256",
        )
        act_signature = signature["Signature"]
        
        return (act_signature)

However, when trying to do the following:

 sign = kms.sign_msg_hash(transaction)
vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
                sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
            )

I get the following error:

ecdsa.util.MalformedSignature: Invalid length of signature, expected 64 bytes long, provided string is 72 bytes long

Now, when trying to use sign[:64] instead, it sometimes works but at other times gives me the following error:

raise SquareRootError("%d has no square root modulo %d" % (a, p))
ecdsa.numbertheory.SquareRootError: 6631794589973073742270549970789085262483305971731159368608959895351281521222 has no square root modulo 115792089237316195423570985008687907853269984665640564039457584007908834671663

I'm not sure if this has anything to do with the encoding of the signature from the KMS or not.

Solution

So after some digging, it turned it out it was a matter of encodings, as the kms sign function returns the signature in DER format.

Hence, what I did was the following:

I took the signature and split it, and got the r and s values as follows:

decoding=ecdsa.util.sigdecode_der(sign,int("fffffffffffffffffffffffffffffffebaaedce6af48a03bbfd25e8cd0364141",16))

I then concatenated the result (note that r and s should be in big-endian order):

new_sign = decoding[0].to_bytes(32, byteorder= 'big') + decoding[1].to_bytes(32, byteorder= 'big')

I then used the new sign:

vks = ecdsa.VerifyingKey.from_public_key_recovery_with_digest(
            new_sign, transaction, curve=ecdsa.SECP256k1, hashfunc=sha256
      )

Another thing I encountered was with kms_client.get_public_key()

First call the method:

kms_pub_key_bytes = kms_client.get_public_key(KeyId=key_id)["PublicKey"]

Then get the last 128 chars which are essentially the r and s values
```
kms_pre_sha3 = kms_pub_key_bytes.hex()[-128:]
```

Then do what you want with the public key.