Java SSL connection error - Invalid keystore format

Question

I'm creating a server and client all-in-one chat application and I'm trying to switch to an SSL connection. I created a keystore.jks and a certificate file (.cer) but now when the program tries to make a connection the acting client throws:

Caused by: java.io.IOException: Invalid keystore format

Here is the code:

System.setProperty("javax.net.ssl.keyStore", "certificates/keystore.jks");
System.setProperty("javax.net.ssl.trustStore", "certificates/certificate.cer");
System.setProperty("javax.net.ssl.keyStorePassword", "password");

if (this.role == ConnectionRole.SERVER) {
    connectingAlert.getJFrame().setVisible(true);
    setupServer();
    do {
        Thread.sleep(10);
    } while (socket == null);
}

if (this.role == ConnectionRole.CLIENT) {
    connectingAlert.getJFrame().setVisible(true);
    setupClient(targetIP);
}

private void setupServer() throws IOException {
    SSLServerSocketFactory sslSrvFact = (SSLServerSocketFactory) SSLServerSocketFactory.getDefault();
    serverSocket = (SSLServerSocket) sslSrvFact.createServerSocket(8080, 1);
    socket = (SSLSocket) serverSocket.accept();
    setupStreams();
}

private void setupClient(String IPAddress) throws IOException {
    SSLSocketFactory sslFact = (SSLSocketFactory) SSLSocketFactory.getDefault();
    socket = (SSLSocket) sslFact.createSocket("localhost", 8080);
    setupStreams();
}

private void setupStreams() throws IOException {
    dataOut = new ObjectOutputStream(socket.getOutputStream());
    dataIn = new ObjectInputStream(socket.getInputStream());
    chatInterface = ChatInterface.getInstance();
}

Answer 1

System.setProperty("javax.net.ssl.trustStore", "certificates/certificate.cer");

The problem is here. A .cer file is not a truststore. You need to import it into a real Java truststore via the keytool with the -trustcacerts option.

BUT it isn't clear why you're using a truststore at all. Are you expecting peers with self-signed certificates to send them to you? Most of the time you should just use the truststore that comes with Java, and don't set javax.net.ssl.trustStore at all.