Does Java1.7 TLS1.2 implementation take dNSName (SAN extension) into account when connecting?

I'm implementing a protocol that specifies TLS1.2 as transport layer and requires client-side server authentication to verify the server's hostname by comparing the hostname value of the connecting client socket to the value indicated by the server in its certificate, namely subjectAltName extension of type dNSName.

I've created a test, put this value in the server's certificate and it seemed to get ignored by the client completely, but I'd like to be sure. Do I have to code this check in an implementation of X509ExtendedTrustManager.checkServerTrusted(X509Certificate[], String, Socket) or can I enable it through some obscure property? The reference guide appears silent on this matter.

The protocol specification (the one I'm implementing) also mentions that wildcards may be used as a prefix of the value in the certificate.

A "*" wildcard character MAY be used as the leftmost name component in the certificate. For example, *.example.com would match a.example.com, foo.example.com, etc., but would not match example.com.

However, when I tried to create such an extension value with keytool, it refused to do so. What's going on?

"C:\Program Files\Java\jdk1.7.0_51\bin\keytool.exe" -genkeypair -alias server -keyalg RSA -validity 365 -ext san=dns:*.example.com -keystore mykeystore ...

keytool error: java.lang.RuntimeException: java.io.IOException: DNSName components must begin with a letter

Solution

Java 7 provides a way to verify the host name automatically on an SSLSocket or SSLEngine, but it is not enabled automatically (this didn't exist in Java 6). The implementation can use the naming specification of LDAP or HTTPS, not the more generic RFC 6125 (at least not yet). In most cases, using this part of the HTTPS specification should be fine for other protocols, certainly better than nothing.

You can use this as follows:

SSLParameters sslParams = new SSLParameters();
sslParams.setEndpointIdentificationAlgorithm("HTTPS");
sslSocket.setSSLParameters(sslParams); // or SSLEngine

You can find references for this:

in the SSLParameters.setEndpointIdentificationAlgorithm(...) documentation,
by searching for "endpoint identification" in the JSSE Reference Guide,
in the JSSE Ref Guide section on X509ExtendedTrustManager ("the X509ExtendedTrustManager class also support algorithm constraints and SSL layer hostname verification"),
by scrolling down to the very last table in the Standard Names section of the Java™ Cryptography Architecture. Standard Algorithm Name Documentation

The second problem you have seems to be an issue with keytool, to generate such a certificate. This doesn't affect how such certificates presented to a Java client are verified. You can use other tools to generate a certificate with a wildcard if necessary.

EDIT:

Note that for this to work, you need to create the SSLSocket with a method that uses String host (so that it knows the host name) or the SSLEngine with SSLContext.createSSLEngine(String peerHost, int peerPort). This is how it knows which hostname to try to match in the certificate. (Using the name is also useful for SNI.)