I have a project that I'm working on, it's a desktop application that is communicating with a server I built over TCP. I want to make the connection secure and encrypted by using SSL (TLS) but I'm not sure, is SSL Certificate required when using SSL in custom protocols? Is it just for HTTPS since it require it?
In addition, I'd like to know what is the role of the SSL Certificate in the encryption process of the stream between the 2 endpoints? Is it just for verifying the 2 endpoints?
Thanks in advance.
... is SSL Certificate required when using SSL in custom protocols?
A certificate is used to identify the server, i.e. to make sure that the client is connecting to the expected server and not to some man in the middle. TLS provides also other methods to do this kind of check, like using a pre-shared secret between client and server, see Wikipedia: TLS-PSK for more.
I'd like to know what is the role of the SSL Certificate in the encryption process of the stream between the 2 endpoints? Is it just for verifying the 2 endpoints?
The primary role of the certificate is to provide authentication. With RSA key exchange the certificate additionally is used to transport the information for the symmetric encryption key from the client to the server - by encrypting the pre-master secret with the servers public key taken from the certificate so that the server can decrypt it with the matching private key. But note that RSA key exchange is considered obsolete and that in modern key exchange methods (Diffie-Hellman) the certificate is not used.