Is encrypting a public key enough security to replace SSL?

Question

I'm looking into implementing OTR (off-the-record) protocol into my de-centralised messaging app (written in Scala with akka and akka-http). I was originally attempting to use SSL, but it seems without shipping my software with a single rootCA, it is pretty difficult (unless I accept all certificates, which poses even more issues).

So using a plain http socket, my plan would be to exchange the public keys for a single session, but encrypt them using a shared password. Each server has a unique ID. This ID is shared to another user that you want to connect and chat with. The other user must enter this ID to establish a connection with your server.

I was thinking that instanceA could use instanceBs serverID to encrypt its public key, then send it over http (with Deniable Authentication). instanceB would then un-encrypt the key and perform DH key-exchange, and do the process back to instanceA.

I would like to understand if there are any potential security risks to this approach vs shipping the software with the same rootCA in every instance.

UPDATE

I seem to be going in circles with this. The correct process seems to be SSL. But:

• The system is de-centralised, meaning each client has its own self-signed CA
• To distribute the rootCA, you must use an encrypted channel over http
• To exchange encryption keys (and prevent MILM attacks), you must sign the key using a RootCA, which needs to be distributed so the client can verify the source of the key.

Doesn't matter what approach I try and take, I end up having to distribute a single rootCA with my software. I don't like this for several reasons:

• The rootCA could be accessed from compiled code
• The rootCA would be shipped with its key, which is password protected. If I really want to make this secure, its best to store it into a keystore. Which has a password around it, pushing the security risk onto the password. Every instance will need the SAME password for the keystore

Please if someone knows an easier way to do this (that does not have these security issues) advise me. I've been going in circles for days

Answer 1

Establishing trust without a single point of failure is a common 'challenge' in decentralized systems, but what you could instead do is use Trust On First Use with Public Key Pinning.

• Instead of distributing a root CA, each instance generates its own self-signed certificate.
• When a user first connects to another instance, they manually verify the fingerprint (e.g., display it via QR code, voice call, or some other 'out-of-band' method). Keep in mind that if this fingerprint is distributed insecurely, it defeats the whole purpose. This is also the main usability downside; depending on the application, manually verifying any fingerprint can be a hassle.
• Once accepted, the public key is "pinned" for future connections.
• If the key changes, the user is alerted to prevent a MiTM attack.