Why is Bouncy Castle DTLS ReceiveRecord larger than the original buffer

Question

I am currently trying to setup a .NET CoAP DTLS Server. My problem is that bouncy castles DTLS management kept crashing just giving me an "internal error (80)". So I downloaded the library and debugged through it, finally finding where it throws. Org.BouncyCastle.Tls.DtlsRecordLayer has the receive method for the package:

internal int Receive(Span<byte> buffer, int waitMillis, DtlsRecordCallback recordCallback)
        {
            long currentTimeMillis = DateTimeUtilities.CurrentUnixMs();

            Timeout timeout = Timeout.ForWaitMillis(waitMillis, currentTimeMillis);
            byte[] record = null;

            while (waitMillis >= 0)
            {
                ... // wait millis timeout checks

                int receiveLimit = m_transport.GetReceiveLimit();
                if (null == record || record.Length < receiveLimit)
                {
                    record = new byte[receiveLimit];
                }

                int received = ReceiveRecord(record, 0, receiveLimit, waitMillis);

                int processed = ProcessRecord(received, record, buffer, recordCallback);
                if (processed >= 0)
                    return processed;

                currentTimeMillis = DateTimeUtilities.CurrentUnixMs();
                waitMillis = Timeout.GetWaitMillis(timeout, currentTimeMillis);
            }

            return -1;
        }

ProcessRecord is what actually decrypts the records and writes it into the buffer. Problem is ReceiveRecord actually returns a bigger byte value, than the buffer is actually sized.

This is the method

        private int ReceiveRecord(byte[] buf, int off, int len, int waitMillis)
        {
            if (m_recordQueue.Available > 0)
                return ReceivePendingRecord(buf, off, len);

            int received = ReceiveDatagram(buf, off, len, waitMillis);
            if (received >= RecordHeaderLength)
            {
                this.m_inConnection = true;

                ... // Epoch checks, to make sure what to read

                int recordHeaderLength = recordEpoch.RecordHeaderLengthRead;
                if (received >= recordHeaderLength)
                {
                    int fragmentLength = TlsUtilities.ReadUint16(buf, off + recordHeaderLength - 2);
                    int recordLength = recordHeaderLength + fragmentLength;
                    if (received > recordLength)
                    {
                        m_recordQueue.AddData(buf, off + recordLength, received - recordLength);
                        received = recordLength;
                    }
                }
            }

            return received;
        }

EDIT I first assumed a fragmentation error, but after closer investigation I realized that the ReceivePendingRecord function is only triggered on handshake, not on the application data packages afterwards. So I assume the record that triggers the exception isn't actually split into fragments, leaving me with no idea at all.

In wireshark I can see that it always crashes when the package is longer than usual, so maybe the problem is somehow still connected to fragmentation

Answer 1

Bouncy castles DTLS methods aren't thread save and never were intended to be. The CoAP library I used didn't account for that, I created an issue and a seperate repository with a fix for my situation at least. https://github.com/srberard/CoAP.NET/issues/2

Keep that in mind if you want to use Bouncy castle for your DTLS Server.