We are building a chat client (something like Whatsapp, etc.) for sending and receiving private messages. For the sake of privacy we decided to encrypt and decrypt the messages upon arrival and sending.
I figured we could use RSA private and public keys for encryption. I understand that we usually need two pairs of keys, one of which residing on the server and one on the phone. This brings up some questions which I'm asking here.
Please feel free to add any useful information for I am a complete newbie in security.
You will need a central server which provides certificates for each user to authenticate users (otherwise you will always be vulnerable to a man in the middle attack). So for two users (Joe and Sally), there will be three certificates: 1) the certificate authenticating your central server, 2) the certificate authenticating Joe and 3) the certificate authenticating Sally.
So when Joe wants to initiate communication with Sally the following should happen:
So why is all of this necessary? First, you need a "trusted" broker (that's the central server). Otherwise you can never be sure who you are communicating with. You may ask: why does Sally need to sign her shared secret key when only Joe can decrypt it anyway (if Sally uses Joe's public key to encrypt)? The reason is that anyone can encrypt a shared secret key via Joe's public key! Therefore if Joe receives an unsigned shared secret key, he may start communicating with a man in the middle who simply generated a shared secret key using Joe's public key. Just as Sally needs some confirmation that Joe is the actual entity trying to communicate with her, Joe needs some assurance that the shared secret key he receives is actually from Sally (hence why she needs to sign with her private key).
RSA is not safe!!! Given enough traffic, it is possible to decrypt messages encoded by RSA. Thus, you should have a protocol in place for periodically updating public/private keys used for authentication. This is simple for users: your central server will hold the public keys for each user. Periodically your server should request that each user generate a new private/public key pair and send the public key to the server to be updated (this communication can be done via the previous private/public key used by each user).
It's a little more "difficult" for the central server itself. If your central server wishes to change its private/public key (authenticating the server), then it needs to communicate with every user to make each user aware of the change. When a user changes his/her private/public key, it's a one-to-one communication (just between the user and the server) but when the server wants to change its private/public key, it must communicate with all users!
Caveat:
I don't claim that what I proposed above is the "canonical" pattern. What I claim, is that if you follow the above, you will will ensure both authenticity and integrity. I.e. you can be guaranteed to be communicating with the entity the central server considers to be "Joe" and "Sally" and you can guarantee that only Joe and Sally can communicate because no one, including the central server, knows the shared secret key used by Joe and Sally to communicate (except for Joe and Sally of course).