Self-signed certificate security for private server/client

Question

I've read a lot that self-signed certificates should never be used in production because of the lack of security but I wonder if it's still a security risk if I'm the only one who're supposed to connect to the server? Is it for some reason easier to crack a self-signed certificate? I'm creating both the server and client application and the only way for someone else to connect is to create their own client. That means that every time I install the client I also have the possibility to add the certificate to the trusted root certificates.

Or is it safer to continue to use my own encryption implementation using RSA/AES on the message level? The reason I want to use SSL instead is that it's much easier to work with, especially when I want to stream media since I don't have to send it in chunks.

Answer 1

I've read a lot that self-signed certificates should never be used in production because of the lack of security ....

Self-signed certificates by itself are not bad and can also used in production if done properly.

A certificates is safe to use if the peer is able to verify it properly. The usual validation is done based on some trusted root CA contained in the browser or operating system. But that a self-signed certificate can not be validated this way does not mean that it cannot be validated at all because:

• You can explicitly add it as trusted to the certificate store of the browser/OS.
• You can make an exception on first use after you've verified that the certificate you get in the browser is actually the one which you know (by comparing the fingerprint, not just the subject).
• If you have your own application you could ship the application so that it (only) trusts this certificate.

Of course explicitly importing the certificate as trusted or making in exception in the browser does not scale well, because it has do be done for each user. And that's the main point of CA-signed certificates: that the certificate gets implicitly trusted because it is signed by someone trusted instead of that each user has to validate and trust the certificate manually. And this is also the only reason you want to use a CA-signed certificate in production. As long as the certificate is properly validated it does not matter if it was self-signed or not.

Or is it safer to continue to use my own encryption implementation using RSA/AES on the message level?

Never run your own crypto unless you really understand what you are doing. In this case SSL provides everything you need but you have to know how to use it properly.