rsa encryption in windows crypto and decryption in openssl

I have a problem in exchanging public key between "OpenSSL" and "Windows CryptoAPI". The public key is exported from OpenSSL in pem format.My program is written in c++. I get the public key and load it by "CryptoAPI". After loading the public key, I encrypt some data and send them to the other application. The other application can not decrypt the received data by own private key. please help me to find the solution.

Source code:

_ServerContextHandle = NULL;
_EncryptionKeyHandle = NULL;

void Initialize(char* inPublicKeyByPemFormat)
{
    HCRYPTPROV serverContextHandle;

    bool result = CryptAcquireContextW(&serverContextHandle, nullptr, nullptr, PROV_RSA_FULL, CRYPT_VERIFYCONTEXT) == TRUE;

    if (result)
    {
        _ServerContextHandle = serverContextHandle;

        byte derPublicKey[2048];
        DWORD derPublicKeyLength = 2048;

        result = CryptStringToBinaryA(inPublicKeyByPemFormat, 0, CRYPT_STRING_BASE64HEADER, derPublicKey, &derPublicKeyLength, nullptr, nullptr) == TRUE;

        CERT_PUBLIC_KEY_INFO* publicKeyInfo = nullptr;
        DWORD publicKeyInfoLength;

        if(result)
        {
            result = CryptDecodeObjectEx(X509_ASN_ENCODING | PKCS_7_ASN_ENCODING, X509_PUBLIC_KEY_INFO, derPublicKey, derPublicKeyLength, CRYPT_ENCODE_ALLOC_FLAG, nullptr, &publicKeyInfo, &publicKeyInfoLength) == TRUE;
        }

        HCRYPTKEY encryptionKeyHandle;

        if(result)
        {
            result = CryptImportPublicKeyInfo(_ServerContextHandle, X509_ASN_ENCODING, publicKeyInfo, &encryptionKeyHandle) == TRUE;
        }

        LocalFree(publicKeyInfo);

        if (result)
        {
            _EncryptionKeyHandle = encryptionKeyHandle;
        }
    }
}


byte* EncryptData(byte* inData, DWORD inDataLength, DWORD* outLength) const
{
    byte* result = nullptr;

    *outLength = 0;

    DWORD length = inDataLength;

    result = CloneByteArray(inData, inDataLength);

    if (!CryptEncrypt(_EncryptionKeyHandle, NULL, TRUE, 0, result, &length, length))
    {
        delet result;

        result = new byte[length];

        CopyByteArray(inData, result, inDataLength);

        *outLength = inDataLength;

        if (!CryptEncrypt(_EncryptionKeyHandle, NULL, TRUE, 0, result, outLength, length))
        {
            delete result;

            result = nullptr;

            *outLength = 0;
        }
    }
    else
    {
        *outLength = length;
    }

    return result;
}

Solution

CryptEncrypt, for reasons that probably made sense to the author at the time, writes the bytes down backwards. Or, rather, it writes them in byte minor/little-endian order whereas almost every other cryptography library (including the Windows CNG BCryptEncrypt and NCryptEncrypt routines) writes them in byte major/big-endian order.

So you need to reverse data coming out of CryptEncrypt, and reverse it going in to CryptDecrypt.

For example, in .NET Core's RSACryptoServiceProvider.Encrypt it calls CapiHelper.EncryptKey which calls CryptEncrypt then Array.Reverse before returning.

The CryptEncrypt documentation has as the last sentence in the Remarks section

The ciphertext is returned in little-endian format.