Call to GSSContext.initSecContext fails intermittently: Receive timed out

Question

I'm writing client-side code for Windows Kerberos authentication with a service (logging code omitted):

System.setProperty("javax.security.auth.useSubjectCredsOnly", "false");
// System.setProperty("sun.security.krb5.debug", "true");
Package thisPkg = AuthHelper.class.getPackage();
String configPath = Util.getConfigPath(thisPkg, "jaas.conf");
System.setProperty("java.security.auth.login.config", "=" + configPath);
GSSManager manager = GSSManager.getInstance();
GSSName peerName = manager.createName(spn, GSSName.NT_HOSTBASED_SERVICE);
GSSContext context = manager.createContext(peerName, null, null,
    GSSContext.DEFAULT_LIFETIME);
context.requestMutualAuth(true); // required
context.requestCredDeleg(true); // required for publish
byte[] serverTokenBytes = new byte[0];
while (!context.isEstablished()) {
  byte[] clientTokenBytes = context.initSecContext(serverTokenBytes, 0,
      serverTokenBytes.length);
  if (clientTokenBytes != null)
    socket.send(createClientMessage(clientTokenBytes));
  if (context.isEstablished()) break;
  Message message = socket.receive();
  String serverToken = message.getFirst("SERVERTOKEN").toString();
  serverTokenBytes = Base64.decodeBase64(serverToken);
}

Where jaas.conf simply contains:

sp {
  com.sun.security.auth.module.Krb5LoginModule required debug=true;
};

I have also set the allowtgtsessionkey registry key as required, and installed JCE Unlimited Strength Jurisdiction Policy Files 7.

The code sometimes works (i.e. mutual authentication is established); however, sometimes it gets stuck for a while at the first call to GSSContext.initSecContext, throwing an exception after about a minute:

Exception in thread "main" GSSException: No valid credentials provided (Mechanism level: Receive timed out)
  ...
Caused by: java.net.SocketTimeoutException: Receive timed out
  ...

When I enable Kerberos debugging output (by uncommenting the second line above), I can see that the protocol sometimes gets stuck at line:

getKDCFromDNS using UDP

A Java Kerberos troubleshooting website suggests that this is an issue with the Kerberos authentication server, but I know that the server is up and running, since we have similar code written in C# (using .NET libraries) that never gets stuck.

Answer 1

It seems like the DNS resolution for the Kerberos authentication server is going through some indirection, which is unreliable. If you specify the server explicitly (somewhere at the beginning of your code), it will bypass that redirection:

System.setProperty("java.security.krb5.realm", "<YOUR_KRB_REALM>");
System.setProperty("java.security.krb5.kdc", "<YOUR_KRB_SERVER_ADDR_OR_IP>");

EDIT: It turns out that communication with Kerberos servers was inherently unreliable due to the protocol using UDP, so it had a high chance of failing for servers that are relatively far away. Windows 8 uses TCP by default; to force TCP on previous versions:

• XP/2000: In HKLM\System\CurrentControlSet\Control\Lsa\Kerberos, set DWORD MaxPacketSize to 1.
• 2003/Vista/7: In HKLM\System\CurrentControlSet\Control\Lsa\Kerberos\Parameters, set DWORD MaxPacketSize to 1.

(Note that the same registry directory also needs DWORD AllowTGTSessionKey set to 1 for Kerberos to work at all.)