How to reliably arrive at a non negative serial number from a SHA1 hash?

Question

I am using CNG to generate serial numbers for certificates. My algorithm takes the common name of the Certificate Authority, appends 10 random bytes and then computes a SHA1 hash of that. The SHA1 hash is consistently 20 bytes in length and I use that as a serial number.

The problem! I don't get reliably positive numbers from that algorithm as far as OpenSSL is concerned. Negative serial numbers cause issues for OpenSSL. So how do I ensure that my 20 bytes are always considered as a 'positive' serial number by OpenSSL?

For example, these serial numbers:

da7db14cbc79401b642fd77806b54e21b012bbe1 f67604707a861fac55fbef8a5571ab8284e761bd 7d8df1b0b62c284ad12fd1eaadfb18dd4c9c91ba 8588ea1034c6c5a23b1f5cf9689e63baf10775a9 169ad01b41f6e5108d64d70bb8de25da475e02b5 89ff69bc06ec5a93c9e11e71a990f7e8ee0a9d3d 6dbd23a8655c9627a8d241d48a909aec7823dc1c

Produce this output when I put them in a Certificate Revocation List, which is unaccepatble:

Revoked Certificates: Serial Number: -25824EB34386BFE49BD02887F94AB1DE4FED441F Revocation Date: Mar 14 21:47:02 2018 GMT Serial Number: -0989FB8F8579E053AA041075AA8E547D7B189E43 Revocation Date: Mar 14 21:47:02 2018 GMT Serial Number: 00 Revocation Date: Mar 14 21:47:02 2018 GMT Serial Number: -7A7715EFCB393A5DC4E0A30697619C450EF88A57 Revocation Date: Mar 14 21:47:02 2018 GMT Serial Number: 169AD01B41F6E5108D64D70BB8DE25DA475E02B5 Revocation Date: Mar 14 21:47:02 2018 GMT Serial Number: -76009643F913A56C361EE18E566F081711F562C3 Revocation Date: Mar 14 21:47:02 2018 GMT Serial Number: 6DBD23A8655C9627A8D241D48A909AEC7823DC1C Revocation Date: Mar 14 21:47:02 2018 GMT

Is there a way to mask off part of one byte to make it always positive? Also, is there a reason why the third serial number is interpreted as zero?

Code sample for generating the serial number:

void CERTSTORE::GetSerialNumber(
eAction eActionTaken,
std::string sCACN, 
std::vector<BYTE> & vSerialNumber
)
{
    NTSTATUS statusBCryptOpenAlgorithmProvider_Hash;
    NTSTATUS statusBCryptOpenAlgorithmProvider_RNG;
    NTSTATUS statusBCryptGenRandom;
    NTSTATUS statusBCryptHash;
    BCRYPT_ALG_HANDLE hRandNumAlg;
    BCRYPT_ALG_HANDLE hHashAlg;
    DWORD dwHash = 20;
    BYTE bRandomBytes[10];
    std::vector<BYTE> vBytesToBeHashed(
        (BYTE)sCACN.c_str(),
        sCACN.length()
    );

    //open algorithm provider to get random number generator
    statusBCryptOpenAlgorithmProvider_RNG = BCryptOpenAlgorithmProvider(
        &hRandNumAlg,
        BCRYPT_RNG_ALGORITHM,
        MS_PRIMITIVE_PROVIDER,
        0
    );

    if (0 != statusBCryptOpenAlgorithmProvider_RNG)
    {
        throw ERRORSTRING(
            eActionTaken,
            eSubAction::eSubAction_OPENALGPROV,
            eMajorErrorCode::eMajorErrorCode_RUNTIMEERROR,
            eMinorErrorCode::eMinorErrorCode_UKNNTSTATUS,
            statusBCryptOpenAlgorithmProvider_RNG
        );
    }

    statusBCryptGenRandom = BCryptGenRandom(
        hRandNumAlg,
        bRandomBytes,
        sizeof(bRandomBytes),
        0
    );

    if (0 != statusBCryptGenRandom)
    {
        BCryptCloseAlgorithmProvider(hRandNumAlg, 0);
        throw ERRORSTRING(
            eActionTaken,
            eSubAction::eSubAction_GENRANDOM,
            eMajorErrorCode::eMajorErrorCode_RUNTIMEERROR,
            eMinorErrorCode::eMinorErrorCode_UKNNTSTATUS,
            statusBCryptGenRandom
        );
    }

    BCryptCloseAlgorithmProvider(hRandNumAlg, 0);

    for (int iRandByteCounter = 0; iRandByteCounter < sizeof(bRandomBytes); iRandByteCounter++)
    {
        vBytesToBeHashed.push_back(
            bRandomBytes[iRandByteCounter]
        );
    }

    statusBCryptOpenAlgorithmProvider_Hash = BCryptOpenAlgorithmProvider(
        &hHashAlg,
        BCRYPT_SHA1_ALGORITHM,
        MS_PRIMITIVE_PROVIDER,
        0
    );

    if (0 != statusBCryptOpenAlgorithmProvider_Hash)
    {
        throw ERRORSTRING(
            eActionTaken,
            eSubAction::eSubAction_OPENALGPROV,
            eMajorErrorCode::eMajorErrorCode_RUNTIMEERROR,
            eMinorErrorCode::eMinorErrorCode_UKNNTSTATUS,
            statusBCryptOpenAlgorithmProvider_Hash
        );
    }

    try
    {
        vSerialNumber.assign(
            dwHash,
            NULL
        );
    }
    catch (std::exception & ex)
    {
        throw ERRORSTRING(
            eActionTaken,
            eSubAction::eSubAction_CRYPTHASH,
            eMajorErrorCode::eMajorErrorCode_RUNTIMEERROR,
            eMinorErrorCode::eMinorErrorCode_MEMORYALLOCATION,
            0
        );
    }

    statusBCryptHash = BCryptHash(
        hHashAlg,
        NULL,
        0,
        &vBytesToBeHashed[0],
        vBytesToBeHashed.size(),
        &vSerialNumber[0],
        dwHash
    );

    if (0 != statusBCryptHash)
    {
        BCryptCloseAlgorithmProvider(hHashAlg, 0);
        throw ERRORSTRING(
            eActionTaken,
            eSubAction::eSubAction_CRYPTHASH,
            eMajorErrorCode::eMajorErrorCode_RUNTIMEERROR,
            eMinorErrorCode::eMinorErrorCode_UKNNTSTATUS,
            statusBCryptHash
        );
    }

    BCryptCloseAlgorithmProvider(hHashAlg, 0);
}

Answer 1

Depending on what API you're using beyond that point, the 20 bytes are either little-endian or big-endian. Since you say the 3rd one came out as negative it appears to be little-endian.

Little Endian Fix:

vSerialNumber[19] &= 0x7F

Big Endian Fix:

vSerialNumber[0] &= 0x7F;

"Eh, just work" Fix:

vSerialNumber[0] &= 0x7F;
vSerialNumber[19] &= 0x7F

You've reduced your entropy by one (or two) bits, but you're still in a safe range.