.net core X509Certificates2 with Fleck WebSocket -> Authentication failed -> An unknown error occurred while processing the certificate

Question

We have an older .net core project using HttpListener and Fleck without ssl. We need ssl now so I started with updating everything. All works well with the webserver but when I try to use the websocket with ssl I get following exception:

Failed to Authenticate System.AggregateException: One or more errors occurred. (Authentication failed, see inner exception.) ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception. ---> System.ComponentModel.Win32Exception (0x80090327): An unknown error occurred while processing the certificate. --- End of inner exception stack trace --- at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm) at System.Threading.Tasks.TaskToApm.End(IAsyncResult asyncResult) at System.Net.Security.SslStream.EndAuthenticateAsServer(IAsyncResult asyncResult) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of inner exception stack trace ---

I used Portable.BouncyCastle 1.8.10 to generate the certificate like this (its the third code I am trying to generate a certificate):

using System;
using System.IO;
using System.Security.Cryptography.X509Certificates;
using Org.BouncyCastle.Asn1.X509;
using Org.BouncyCastle.Crypto;
using Org.BouncyCastle.Crypto.Generators;
using Org.BouncyCastle.Crypto.Operators;
using Org.BouncyCastle.Crypto.Prng;
using Org.BouncyCastle.Math;
using Org.BouncyCastle.Pkcs;
using Org.BouncyCastle.Security;
using Org.BouncyCastle.Utilities;
using Org.BouncyCastle.X509;


namespace WebServer
{
    public static class CertificateHelper
    {
        private const string DefaultName = "X509Cert2.PFX";

        // Source: Přemysl Šťastný https://stackoverflow.com/questions/51684883/generate-self-signed-x509certificate2-using-net-standard-2-0
        public static X509Certificate2 GenerateCertificate(string subjectName = "CN=root ca", int keyStrength = 4096)
        {
            // Generating Random Numbers
            var randomGenerator = new CryptoApiRandomGenerator();
            var random = new SecureRandom(randomGenerator);

            // The Certificate Generator
            var certificateGenerator = new X509V3CertificateGenerator();

            // Serial Number
            var serialNumber = BigIntegers.CreateRandomInRange(BigInteger.One, BigInteger.ValueOf(Int64.MaxValue), random);
            certificateGenerator.SetSerialNumber(serialNumber);

            // Signature Algorithm
            const string signatureAlgorithm = "SHA256WithRSA";

            // Issuer and Subject Name
            var subjectDN = new X509Name(subjectName);
            var issuerDN = subjectDN;
            certificateGenerator.SetIssuerDN(issuerDN);
            certificateGenerator.SetSubjectDN(subjectDN);

            // Valid For
            var notBefore = DateTime.UtcNow.Date;
            var notAfter = notBefore.AddYears(20);

            certificateGenerator.SetNotBefore(notBefore);
            certificateGenerator.SetNotAfter(notAfter);

            // Subject Public Key
            AsymmetricCipherKeyPair subjectKeyPair;
            var keyGenerationParameters = new KeyGenerationParameters(random, keyStrength);
            var keyPairGenerator = new RsaKeyPairGenerator();
            keyPairGenerator.Init(keyGenerationParameters);
            subjectKeyPair = keyPairGenerator.GenerateKeyPair();

            certificateGenerator.SetPublicKey(subjectKeyPair.Public);

            // Generating the Certificate
            var issuerKeyPair = subjectKeyPair;

            ISignatureFactory signatureFactory = new Asn1SignatureFactory(signatureAlgorithm, issuerKeyPair.Private, random);

            // selfsign certificate
            var certificate = certificateGenerator.Generate(signatureFactory);

            // in-memory PFX stream
            var pkcs12Store = new Pkcs12Store();
            var certEntry = new X509CertificateEntry(certificate);
            pkcs12Store.SetCertificateEntry(subjectName, certEntry);
            pkcs12Store.SetKeyEntry(subjectName, new AsymmetricKeyEntry(subjectKeyPair.Private), new[] {certEntry});
            X509Certificate2 keyedCert;
            using (MemoryStream pfxStream = new MemoryStream())
            {
                pkcs12Store.Save(pfxStream, new char[0], new SecureRandom());
                pfxStream.Seek(0, SeekOrigin.Begin);
                keyedCert = new X509Certificate2(pfxStream.ToArray(), string.Empty, X509KeyStorageFlags.Exportable);
            }

            return keyedCert;
        }

        public static void Save(this X509Certificate2 cert, string outputFileName)
        {
            var bytes = cert.Export(System.Security.Cryptography.X509Certificates.X509ContentType.Pkcs12);
            File.WriteAllBytes(outputFileName, bytes);
        }

        public static X509Certificate2 SaveNewX509Certificate2(string fileName = DefaultName)
        {
            var cert = GenerateCertificate();
            cert.Save(DefaultName);
            return cert;
        }

        public static string GetInstallX509Certificate2(string fileName = DefaultName)
        {
            if (!File.Exists(fileName))
            {
                var cert = SaveNewX509Certificate2(fileName);
                X509Store store = new X509Store(StoreName.Root, StoreLocation.LocalMachine);
                store.Open(OpenFlags.ReadWrite);
                if (!store.Certificates.Contains(cert))
                {
                    store.Add(cert);
                }

                store.Close();
                return fileName;
            }
            else
            {
                return fileName;
            }
        }
    }
}

The CertificateHelper.GetInstallX509Certificate2() method is what I run in the end.

And then I start the Fleck 1.2.0 websocket like this (SSLCertificatePath is a string holding the file path):

var server = new WebSocketServer(wsLocation)
{
    SupportedSubProtocols = webSocketHandlers.Keys
};
if (SSLCertificatePath != null)
{
    server.EnabledSslProtocols = System.Security.Authentication.SslProtocols.Tls12;
    server.Certificate = new X509Certificate2(SSLCertificatePath);
}
server.Start(...);

Server starts fine but as soon as I try to connect to the websocket, the above exception is thrown. Honestly I have no clue what I am doing, never worked with certificates before.

Answer 1

Didn't really find a solution to the problem but I resolved it by integrating websocket support into our httplistener instead of using a library. The http listener was already working with ssl so websockets worked automagically.