Search code examples
javassljssesslengine

SSLException: Invalid padding Unwrapping TLS Application Data

Several months ago I developed an SSL web server using NIO and the SSLEngine. GET requests work great as do small POST requests (under ~10KB). However, I'm getting sporadic SSLException exceptions, when I POST anything larger than that.

For example, if I POST a small image (~16KB), I can see 3 TLS records representing application data. The first is the HTTP header, and the other two contain the payload. Here's a sense of how small the packets are:

1: 613 bytes
2: 16341 bytes
3: 549 bytes

In my code, I call the SSLEngine to unwrap/decrypt the TLS records (application data). Frequently, the SSLEngine chokes when it tries to unwrap the second record. Here's the error:

javax.net.ssl.SSLException: Invalid padding
    at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.fatal(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.readNetRecord(Unknown Source)
    at com.sun.net.ssl.internal.ssl.SSLEngineImpl.unwrap(Unknown Source)
    at javax.net.ssl.SSLEngine.unwrap(Unknown Source)
    at javaxt.http.servlet.HttpServletRequest.getApplicationData(HttpServletRequest.java:1793)
    ...

    Caused by: javax.crypto.BadPaddingException: Invalid TLS padding: 199
    ...

Here's the code that's throwing the error. 

private byte[] getApplicationData() throws IOException {

  //Read the next 5 bytes from the socket channel. This should contain TLS
  //record information.
    if (recordHeader==null){
        recordHeader = ByteBuffer.allocateDirect(5);
        read(recordHeader);
    }

  //Verify that the message contains application data
    recordHeader.rewind();
    if(recordHeader.get()!=23) throw new IOException();

  //Get the length of the TLS record
    recordHeader.position(3);
    int recordLength = Integer.parseInt(getHex(recordHeader) + getHex(recordHeader), 16);
    recordHeader.rewind();

  //Read the TLS record
    ByteBuffer recordData = ByteBuffer.allocateDirect(recordLength);
    read(recordData);
    recordData.rewind();

  //Merge the TLS header and record data into a single buffer
    ByteBuffer tlsRecord = ByteBuffer.allocateDirect(recordLength+recordHeader.capacity());
    tlsRecord.put(recordHeader);
    tlsRecord.put(recordData);
    tlsRecord.rewind();

  //Decrypt the application data
    ByteBuffer output = ByteBuffer.allocateDirect(tlsRecord.capacity());
    SSLEngineResult serverResult = sslEngine.unwrap(tlsRecord, output);
    runDelegatedTasks(serverResult, sslEngine);


    byte[] arr = new byte[output.position()];
    output.rewind();
    output.get(arr);


  //Clean up
    recordHeader.clear();
    recordData.clear();
    tlsRecord.clear();
    output.clear();
    recordHeader = recordData = tlsRecord = output = null;


  //Return array
    return arr;
}

The SSLException is thrown at this line:

SSLEngineResult serverResult = sslEngine.unwrap(tlsRecord, output);

This error occurs sporadically. Sometimes I see it, sometimes I don't. Oftentimes, if I do see the error and simply re-POST the data (e.g. refresh the browser), everything works great (no errors).

I am not an SSL expert by any means so I don't know how best to debug the issue. I'm pretty sure I'm calling the unwrap method correctly. Is the second record somehow corrupt? Has the cypher changed and I need to re-initiate a handshake? What do I do when I encounter this error?

Couple other points:

  • I'm using Sun/Oracle JDK 6 and Java Secure Socket Extension (JSSE) that comes bundled with it.
  • I'm seeing this error a lot when with Mobile Safari, iOS 6. I rarely have this issue with FireFox.

Thanks in advance!

Solution

  • After a long hiatus, I finally got around to resolving this issue. The problem was that when I was reading the TLS record, I was assuming that my read() method was returning all the requested bytes (i.e. the recordData ByteBuffer was full). That was not true. As a result, the sslEngine.unwrap() call was bombing out.

    To resolve the issue, I simply replaced this:

      //Read the TLS record
    ByteBuffer recordData = ByteBuffer.allocateDirect(recordLength);
    read(recordData)

    With this:

      //Read the TLS record
    ByteBuffer recordData = ByteBuffer.allocateDirect(recordLength);
    int ttl = read(recordData);
    if (ttl<recordLength){
        while (ttl<recordLength){
            recordData.position(ttl);
            ByteBuffer tmp = ByteBuffer.allocateDirect(recordLength-ttl);
            ttl += read(tmp);
            recordData.put(tmp);
        }
        recordData.rewind();
    }

    Once the recordData ByteBuffer was filled, the SSLEngine unwrap() method worked flawlessly.