How can I connect a C# SSLStream client to an OpenSSL server?

Question

I am currentry writing a C# client (with Mono using SSLStream) for an already existing OpenSSL server written in C++. I know that in theory, they should be able to communicate, but I always get an Exception:

Authentication or decryption has failed.

Which takes place in Mono.Security.Protocol.Tls.SslStreamBase.AsyncHandshakeCallback when I invoke sslStream.AuthenticateAsClient(ServerName);

I do not need client authentication, so i don't use any client certificate, and i can't load my server certificate into a client's trust store, as I can't use the command line on the client side.

On the server, I seem to receive some random data, which should not happen, as I never invoke sslStream.Write() in the client code. Maybe I don't completely understand how the handshake happens. I have an OpenSSL C++ client that works perfectly though.

Does someone know How I could get this working?

I am using, on the server side, a self-signed certificate generated by OpenSSL, which i converted to DER format.

How I proceed:

Server side

1. Initialize OpenSSL
2. Load server certificate (DER) and private key (PEM)
3. Open a non-SSL listening socket
4. As a client connects, I open a new socket which I connect to an SSL_Bio and accept the connection
5. Read the SSL socket for incoming data.

Client side

I am trying to connect using a temporary client adapted from the one in this topic.

1. Open a new TCPClient
2. Link a new SSLStream to it with the same validation callback as in the thread above;
3. Calling AuthenticateAsClient(ServerName) which brings this exception.
4. Invoke sslStream.Write but the execution never gets here.

Any help would be greatly appreciated as I am not sure to understand how sslStream internally works.

Answer 1

Ok, got it. In fact, I was setting up the BIO and the SSL variables in the wrong order. You have to create and attach your BIO to your SSL BEFORE you actually call SSL_Accept in OpenSSL, otherwise the authentication cannot complete, which seems logical.

Also, I was using SSLStream in C# on client side to send my messages, while I was using BIO_read and BIO_write on the server Side with OpenSSL.

This could not work, because BIO_read and BIO_write are no equivalent to SSL_read and SSL_write. These two functions only read and write on the underlying socket, without any encryption or decryption, making the SSL handshake almost useless (aside from checking the identity of the other end of the line).

This explains why I was not able to engage communication as the two ends did not speak the same language: one sent unencrypted messages while the other expected SSL-wrapped messages and the other way round.

The SSL_read and SSL_Write functions take care of the encryption and decryption, so they are the ones to use here. Now it works perfectly.

If you need any further details about how I did that, please comment this answer.