I need an encrypted UDP connection in Java. I know about DTLS, but it is problematic in Java. So I would much prefer to do my own encryption using JKS or JCE. Why UDP? Some lost packets or reordering doesn't concern me, but latency does. So far I have this concept:
The server creates a temporary symmetric encryption key (unique for the session), encrypts it with the public key of the client (asymmetric encryption) and sends it to the him. The rest of the session they communicate with datagrams encrypted using the symmetric key only.
What are the disadvantages of using this approach as opposed to DTLS? Speed? Security?
That almost sounds like you are trying to re-invent DTLS. You are saying DTLS is problematic, but why is this?
In theory your logic is sound though. It reminds me of DTLS with RSA key exchange:
ClientRandom)ServerRandom)Pre-Master Secret encrypted with server public key)Now both parties have: ClientRandom, ServerRandom which are public knowledge and Pre-Master Secret which has been shared over the network only with asymmetric encryption. Only the owner of the Certificate knows the private key required to decrypt Pre-Master Secret from the network communication so it is important that the Client verifies the Certificate the server sent.
DTLS determines the method for generating Master Secret from Pre-Master Secret, ClientRandom and ServerRandom and the symmetric encryption/MAC keys from the Master Secret. DTLS uses Pre-Master Secret instead of directly sharing the Master Secret for modularity. From security stand point it should be equally secure to share Master Secret directly. The ClientRandom and ServerRandom are technically used as salt in the key generation. This guarantees that both participants get a say in the cryptographic security of the salt.
Once the symmetric encryption keys are generated, the participants further verify that they can communicate using them before concluding the handshake.
When using CBC the encrypted packets themselves are in the form of:
[ IV ] + encrypted( [ Data blocks ] [ HMAC ] [ Padding ] )
Sending the Initialization vector with every packet adds some overhead to the transmission, but defends against some chosen plaintext attacks. However this is only one of the security measures against different attacks. There are a lot of different nuances in effective (D)TLS implementation: Ignoring or hiding error conditions, making sure operations take constant time even if they fail, etc.
So while the secure idea of DTLS can be summed up with "Share public key to one participant. Transfer symmetric key with asymmetric encryption. Encrypt the communication with the symmetric key", there's a lot more small details that makes it secure. It's these small details that usually fail when people invent their own security measures.
Effective MITM attacks are reduced by verifying the certificate - of course in the above handshake only the client verifies the server identity. If you need for the server to verify the client identity the client would need to send its own certificate as well, etc.