How can a process authenticate and communicate securely with another process on the same host

I was trying to do this with Java RMI over SSL, but later I discovered that SSL certificates will provide host level authentication and NOT process level authentication.

Also, I was storing the keystore's password in configuration; so the certificates can be used by another attacker process and it can get authenticate.

Solution

An X.509 certificate used for SSL/TLS could potentially be used to identify something else than a host name (this is already typically done for client certificates).

There are two types of verification involved when establishing an SSL/TLS connection to a server:

The certificate verification itself: this verifies that the certificate is trusted and valid for the required purpose at the time of use, usually with a PKI as described in RFC 3280/RFC 5280.
The host name verification: once it trusts the certificate to be genuine, the client checks that it's for the server it was looking for. This is protocol specific (e.g. RFC 2818, Section 3.1 for HTTPS), but has been generalised for most protocols in RFC 6125. (This is similar to checking that the picture on a passport matches the name in front of you, instead of just accepting any valid passport.)

By default, Java's SSLSockets don't perform the second step unless you add something to do it. (In Java 7, some new SSL parameters allow you to do so within the trust manager, but only for specific protocols.)

What you'd need is to find a way to define how you want to identify your other applications and processes, using something else than the host name, issues certificates with these naming conventions, and have your client application check this.

You should be able to implement your own identity verification mechanism within anSSLSocketFactory, before returning the sockets in each method, an use that factory for your RMI application, as described here: https://blogs.oracle.com/lmalventosa/entry/using_the_ssl_tls_based1