We have a client application on a Windows 7 SP1 VM with the appropriate hotfixes and registry settings to enable TLS 1.2 communication. We have a server application on a Windows Server 2019 Datacenter VM. The client and server establish a TLS 1.2 session (according to QueryContextAttributes) and the negotiated stream trailer size is 48 bytes (again, according to QueryContextAttributes).
When either the client or server calls DecryptMessage with four buffers (one SECBUFFER_DATA and three SECBUFFER_EMPTY), the output buffers are of type SECBUFFER_STREAM_HEADER, SECBUFFER_DATA, SECBUFFER_STREAM_TRAILER, and SECBUFFER_EXTRA.
In spite of the negotiated stream trailer size being some negotiated number of bytes (eg. 48 as above), the SECBUFFER_STREAM_TRAILER seems to always be smaller by a few bytes and the SECBUFFER_EXTRA points to the first byte after the real end of the stream trailer.
For example, if the negotiated stream trailer size was 48 bytes and we decrypted a message packet with a SECBUFFER_STREAM_TRAILER buffer with a cbBuffer of 45 bytes, then the SECBUFFER_EXTRA buffer would be present and point to the stream trailer's buffer + 45 and would have a cbBuffer of 3.
This seems extremely odd to me. In an extremely technical sense, it is valid (the extra buffer is simply used to point to data which was not consumed by the previous call to DecryptMessage, and those 3 bytes in the example above were not). We've resolved the issue by calculating whether the extra buffer is within the negotiated stream trailer block and shifting the extra buffer pointer past those bytes if it is, but it seems strange that the security provider would even report those bytes as "extra data" or that a message packet would use fewer bytes for its stream header and/or stream trailer.
Notably, this behavior does not seem to be present when both the client and server are on more modern operating systems than Windows 7 (tested on Windows Server 2012 R2 Standard, Windows Server 2019 Datacenter, and Windows 10).
The negotiated stream sizes represent the maximum possible value of the header and trailer. The actual values may be less. When the client was sending TLS application data messages, Schannel would sometimes construct a trailer that was smaller than the negotiated size but our application would always send the [maximum header size] + [application data size] + [maximum trailer size] bytes which would result in the server receiving extra null bytes. Schannel's DecryptMessage would then report those extra bytes as extra data but from our perspective that data was "inside" the trailer.
As mentioned previously, we resolved the issue by calculating whether the extra buffer was inside what we thought was the trailer and would move the extra data pointer to point to just after those null bytes (if there was any data after those null bytes) which made everything else work.
However, properly implemented clients would fail to communicate with the server over TLS 1.0 and TLS 1.1 sessions. This led me to fix the bug in our encryption logic which would result in our applications sending extra null bytes and the bug in our decryption logic which would result in applications invalidly skipping over those (bad) null bytes.