Why is CDOSYS sending me a type 3 message with a 56 byte response?

Question

I have some really old code that I'd like to get working with the CDO.Message object. Many years ago, I think this code did work (Windows 98?), but then it didn't work starting with Windows XP (now I'm on Windows 10).

My code first checks sResponse2.cLength, and if it's 24, then it performs the NTLM calculation. Failing that, if sResponse1.cLength is also 24, then it performs the LM calculation instead. That was all I ever had implemented.

Today, with the CDO.Message testing, it took the LM code path, and the hash that my code calculated did not match what CDO.Message provided. That's another problem.

What I'm more interested in right now is why sResponse2.cLength is 56. What does a value of 56 indicate? Given that value and the flags, should I be testing the type 3 message response using a different algorithm?

When I send the type 2 message to the client, I'm only specifying these two flags:

#define F_NEGOTIATE_OEM             0x00000002
#define F_NEGOTIATE_NTLM            0x00000200

Answer 1

The response was greater than 24 bytes because it was an NTLMv2 response. The 56 bytes include the 16-byte HMAC-MD5 hash, a blob signature, two "reserved" fields, an 8-byte time stamp, an 8-byte client nonce, and variable length target data (12 bytes of overhead in this case). 56 bytes may be the minimum possible size with an empty target server/domain string.

Had it been 24 bytes, it could have instead been an NTLMv2 Session Response, which is different.

Details can be found here: http://davenport.sourceforge.net/ntlm.html

I've created the following code to parse the NTLMv2 response:

struct TYPE3_BLOCK
{
    const BYTE* pcbHash;        // 16 bytes

    const BYTE* pcbBlockPtr;    // Points to the block after the hash
    DWORD cbBlockPtr;

    DWORD dwSignature;
    FILETIME ftStamp;
    const BYTE* pcbNonce;       // 8 bytes
    DWORD dwReserved1;
    DWORD dwReserved2;
    const BYTE* pcbTarget;      // Remainder, less dwReserved2
    DWORD cbTarget;
};

template <typename T>
BOOL TConsumePtr (const BYTE*& pcbData, DWORD& cbData, T** pptPtr, DWORD cbPtr)
{
    if(cbData >= cbPtr)
    {
        *pptPtr = reinterpret_cast<T*>(pcbData);
        pcbData += cbPtr;
        cbData -= cbPtr;
        return TRUE;
    }

    return FALSE;
}

template <typename T>
BOOL TConsumeData (const BYTE*& pcbData, DWORD& cbData, T* ptPtr)
{
    if(cbData >= sizeof(T))
    {
        CopyMemory(ptPtr, pcbData, sizeof(T));
        pcbData += sizeof(T);
        cbData -= sizeof(T);
        return TRUE;
    }

    return FALSE;
}

BOOL CrackType3Response (const BYTE* pcbResponse, DWORD cbResponse, __out TYPE3_BLOCK* pBlock)
{
    if(TConsumePtr(pcbResponse, cbResponse, &pBlock->pcbHash, 16))
    {
        pBlock->pcbBlockPtr = pcbResponse;
        pBlock->cbBlockPtr = cbResponse;

        if(TConsumeData(pcbResponse, cbResponse, &pBlock->dwSignature) &&
            TConsumeData(pcbResponse, cbResponse, &pBlock->dwReserved1) &&
            TConsumeData(pcbResponse, cbResponse, &pBlock->ftStamp) &&
            TConsumePtr(pcbResponse, cbResponse, &pBlock->pcbNonce, 8))
        {
            pBlock->cbTarget = cbResponse - sizeof(pBlock->dwReserved2);

            return TConsumePtr(pcbResponse, cbResponse, &pBlock->pcbTarget, pBlock->cbTarget) &&
                TConsumeData(pcbResponse, cbResponse, &pBlock->dwReserved2);
        }
    }

    return FALSE;
}