How can I securely authenticate to a server socket with java?

Question

I have written a java program, where a client connects to a server via socket (not SSLSocket!). First the client creates an RSA-keypair and then sends the public key to the server. Then the server responds with a public key, which the client uses to create the RSA-encoded login password for authentication.

Is this a safe way or, if not, how can I ensure the security of the password?

Answer 1

You would almost never implement a handshake protocol yourself unless you were an expert in cryptography.

Instead, since I can presume you're using a protocol like TCP, UDP, or HTTP, you can use libraries like Netty or just the built in TLS/SSL support.

Netty SSL Example

SSLSocket (i know you said you didnt use it) class for no 3rd party usage.