VB code using advapi32.dll cryptverifysignature works on Vista SP2 not on Windows 10 64-bit

Question

I have been trying to troubleshoot why an element of an application written and compiled in VB6 on a Vista SP2 machine works perfectly on that machine but not on a 64-bit Windows 10 machine. To improve debugging capabilities I replicated the problem coding in Excel VBA. Works on the Vista machine but not on the Windows 10 machine.

The CryptoAPI calls are all declared thusly:

Private Declare Function CryptVerifySignature _
Lib "advapi32.dll" _
    Alias "CryptVerifySignatureA" ( _
        ByVal hHash As Long, _
        pbSignature As Byte, _
        ByVal dwSigLen As Long, _
        ByVal hPubKey As Long, _
        ByVal sDescription As String, _
        ByVal dwFlags As Long _
            ) As Long

The part that is failing on the Windows 10 machine is highlighted by train tracks above and below in the following:

Private Function SignValidate(ByRef abData() As Byte, _
                        ByRef abSigned() As Byte, _
                        Optional bSigned As Boolean = True) As Long
Dim hHash As Long
Dim lngReturnValue As Long
Dim lngSigLen As Long
Dim abText() As Byte
Dim strTxt As String
Dim lngW As Long
Dim lngX As Long
Dim lngY As Long

Dim abHashVal() As Byte

SignValidate = -1
ReDim abText(UBound(abData))
abText = abData

'Create a hash object to sign/validate
lngReturnValue = CryptCreateHash(hCryptProv, CALG_SHA, 0, 0, hHash)
If lngReturnValue = 0 Then
    'Set_locale regionalSymbol
    Err.Raise Err.LastDllError, , "DLL error code shown above. Could not create a Hash Object (CryptCreateHash API)"
End If
'Hash the data
lngW = UBound(abText) + 1
lngReturnValue = CryptHashData(hHash, abText(0), lngW, 0)
If lngReturnValue = 0 Then
    'Set_locale regionalSymbol
    Err.Raise Err.LastDllError, , "DLL error code shown above. Could not calculate a Hash Value (CryptHashData API)"
End If

If bSigned Then
    'release old key pair handle
    If hKeyPair <> 0 Then CryptDestroyKey hKeyPair
    'get a handle to the signature key pair
    lngReturnValue = CryptGetUserKey(hCryptProv, AT_SIGNATURE, hKeyPair)
    If lngReturnValue = 0 Then
        'Set_locale regionalSymbol
        Err.Raise Err.LastDllError, , "DLL error code shown above. Could not obtain key pair"
    End If
    'Determine the size of the signature
    lngReturnValue = CryptSignHash(hHash, AT_SIGNATURE, 0, 0, vbNull, lngSigLength)
    If lngSigLength > 0 Then ReDim abSig(lngSigLength - 1)
    'Sign the hash object
    lngReturnValue = CryptSignHash(hHash, AT_SIGNATURE, 0, 0, abSig(0), lngSigLength)
    If lngReturnValue = 0 Then
        'Set_locale regionalSymbol
        Err.Raise Err.LastDllError, , "DLL error code shown above. Could not sign the hash"
    End If
    ' the signature is now available
    ' size returned array to signature length
    ReDim abSigned(UBound(abSig))
    ' return the signature to the calling procedure
    abSigned = abSig
    SignValidate = 0
Else
    lngSigLength = UBound(abSigned) + 1
    ReDim abSig(UBound(abSigned))
    abSig = abSigned ' load the Signature array

'========================================================
    'this is the line where the actual validation is done
    lngReturnValue = CryptVerifySignature(hHash, abSig(0), lngSigLength, hKeyPair, 0, 0)
'========================================================
    If lngReturnValue = 0 Then 'some error occurred
        SignValidate = Err.LastDllError
    Else
        SignValidate = 0
    End If
End If
End Function

The Windows 10 machine fails on the highlighted call to CryptVerifySignature and returns an Err.LastDllError equal to NTE_BAD_SIGNATURE. The Vista machine validates the signature fine.

I have spent days researching what might be happening here. All to no avail. Any pointers gratefully received

Answer 1

After much frustration and fruitless research I've eventually discovered what the problem was. Along the way I discovered that the problem originated in another part of the code altogether. I also discovered that the problem manifested itself on Windows 10 32 bit as well - so not a 64 bit problem.

An incorrectly populated dwflags argument to an earlier call to CryptImportKey didn't seem to prevent the call to CryptVerifySignature succeeding under Vista 32 bit, even though the call to CryptImportKey, on investigation, had failed. Once the dwflags argument of CryptImportKey was corrected, to CRYPT_EXPORTABLE Or CRYPT_NO_SALT, it succeeded and the subsequent call to CryptVerifySignature succeeded under all alternative operating system / bit number combinations I was able to test.

Apologies and thanks in equal measure to all those who have sought to help in this matter. Until the next time.