NTLM authentication fails after type 2 response

Question

I have a site with SSO, which is using NTLM authentication, which was working perfectly so far on XP and win 7 (32bit), but recently my company decided to use win 7 (64bit) computers as well. On these PCs, the handshake ends after type 2 message, and tomcat returns 401. I have no clue how to investigate, maybe someone here can give me some tips.

This is in the servlet's doPost method:

        try {
        // NTLM Handshake
        // 1: Client --> Server | GET ...
        // 2: Client <-- Server | 401 Unauthorized/WWW-Authenticate: NTLM

        String auth = request.getHeader("Authorization");
        if (auth == null) {
            response.setStatus(response.SC_UNAUTHORIZED);
            response.setHeader("WWW-Authenticate", "NTLM");
            response.flushBuffer();
            return;
        }
        if (auth.startsWith("NTLM ")) {
            byte[] msg = new sun.misc.BASE64Decoder().decodeBuffer(auth
                    .substring(5));
            int off = 0, length, offset;

            // 3: Client --> Server | GET .../Authorization: NTLM
            // <base64-encoded type-1-message>
            if (msg[8] == 1) {
                // 4: Client <-- Server | 401 Unauthorized/WWW-Authenticate:
                // NTLM <base64-encoded type-2-message>
                byte z = 0;
                byte[] msg1 = { (byte) 'N', (byte) 'T', (byte) 'L',
                        (byte) 'M', (byte) 'S', (byte) 'S', (byte) 'P', z,
                        (byte) 2, z, z, z, z, z, z, z, (byte) 40, z, z, z,
                        (byte) 1, (byte) 130, z, z, z, (byte) 2, (byte) 2,
                        (byte) 2, z, z, z, z, z, z, z, z, z, z, z, z };

                response.setHeader("WWW-Authenticate", "NTLM "
                        + new sun.misc.BASE64Encoder().encodeBuffer(msg1));
                //response.sendError(response.SC_UNAUTHORIZED);
                response.setStatus(response.SC_UNAUTHORIZED);
                response.flushBuffer();
                return;
            }
            // 5: Client --> Server | GET .../Authorization: NTLM
            // <base64-encoded type-3-message>
            else if (msg[8] == 3) {
                off = 30;

                length = msg[off + 17] * 256 + msg[off + 16];
                offset = msg[off + 19] * 256 + msg[off + 18];
                String remoteHost = new String(msg, offset, length);

                length = msg[off + 1] * 256 + msg[off];
                offset = msg[off + 3] * 256 + msg[off + 2];
                String domain = new String(msg, offset, length);

                for (int i = 0; i < domain.length(); i += 2)
                    loginDomain += domain.charAt(i);

                length = msg[off + 9] * 256 + msg[off + 8];
                offset = msg[off + 11] * 256 + msg[off + 10];
                String username = new String(msg, offset, length);

                for (int i = 0; i < username.length(); i += 2)
                    loginUser += username.charAt(i);
            }
        }
    } catch (Exception se) {
        log.error(loginUser + ", NTLM handshake exception:", se);
    }

Answer 1

It was a policy issue on the 64 bit PCs as NTLMv2 was forced. Solution is to disable these setting in security policy.

• Run "secpol.msc" This should bring up a window titled "Local Security Policy"

• Security Settings > Local Policies >Security Options > "Network Security: Minimum session security for NTLM SSP based …"

There are two similar entries which begins like this, open and uncheck "Require NTLMv2 session security".

At least it turned out we are using NTLMv1. :|