Strange crash with an self hosted owin webapi

Question

I have a console app the self hosts an owin webapi that converts a pdf to pdfa and another console app that calls the webapi.

The problem is a bit strange. I can call the webapi twice in a row with a file that is 33KB in size but if I call it twice with a 2.6mb file, from the server point of view it returns the converted file data both times, but the second time there is a short delay before the the server hard exits without reporting an error and the client reports the following nested exceptions.

1. "HttpRequestException: Error while copying content to a stream."
2. "Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host"
3. "An existing connection was forcibly closed by the remote host" 

Now for the server side code

public class Startup
{
    public void Configuration(IAppBuilder app)
    {
        // Configure Web API for self-host. 
        HttpConfiguration config = new HttpConfiguration();
        config.MapHttpAttributeRoutes();
        config.Routes.MapHttpRoute(
            name: "DefaultApi",
            routeTemplate: "api/{controller}/{id}",
            defaults: new { id = RouteParameter.Optional }
        );

        app.UseCors(CorsOptions.AllowAll);
        app.UseWebApi(config);

        JsonConvert.DefaultSettings = () => new JsonSerializerSettings
        {
            NullValueHandling = NullValueHandling.Ignore,
            Formatting = Formatting.None,
        };
    }
}

And the self hosted server is started with the following

WebApp.Start<Startup>(url);

The controller

public class ConvertPdfByteArrayToPdfaController : ApiController
{
    public ConvertPdfByteArrayToPdfaResponse Post([FromBody]ConvertPdfByteArrayToPdfaRequest request)
    {
        try
        {                    
            //lots of work here

            jsonLight = result.GetJsonLight();
            return result;
        }
        catch (Exception ex)
        {
            throw logger.LogExceptionForReThrow("There was a problem trying to convert the PDF to PDFa", ex);
        }
        finally
        {    
            logger.End("Returned", $"Returned: {jsonLight}");
        }
    }
}

This completes fine with the return and logger succeeding both times.

Now for the client side

private HttpClient GetWebApiPdfClient(string serverHostName, int portNumber)
{
    var client = new HttpClient();
    client.BaseAddress = new Uri($"http://{serverHostName}:{portNumber}/");
    client.DefaultRequestHeaders.Accept.Clear();
    client.DefaultRequestHeaders.Accept.Add(MediaTypeWithQualityHeaderValue.Parse("application/json"));
    client.Timeout = TimeSpan.FromHours(2);
    client.MaxResponseContentBufferSize = 1000000000;
    return client;
}


public ConvertPdfByteArrayToPdfaResponse ConvertPdfByteArrayToPdfa(byte[] byteArray, bool certify)
{
    var request = new ConvertPdfByteArrayToPdfaRequest { PdfByteArray = byteArray, Certify = certify };
    return PostObject<ConvertPdfByteArrayToPdfaResponse>("api/convertPdfByteArrayToPdfa", request);
}

private T PostObject<T>(string endpoint, object data)
{
    using (var client = GetWebApiPdfClient())
    {
        var json = JsonConvert.SerializeObject(data);
        var content = new StringContent(json, Encoding.UTF8, "application/json");

        var response = client.PostAsync(endpoint, content).Result;  //Fails the second time here on a 2.6mb file but works fine for first file 2.6mb file
        var responseText = response.Content.ReadAsStringAsync().GetAwaiter().GetResult();

        if (response.StatusCode != HttpStatusCode.OK)
            throw new Exception($"Expected returnCode:{HttpStatusCode.OK} but received {response.StatusCode}");
        if (response.Content.Headers.ContentLength > 0)
            return JsonConvert.DeserializeObject<T>(responseText);

        throw new Exception($"response content length was 0");
    }
}

The calling code

            var webApiPdfClient = new WebApiPdfClient();
            var response = webApiPdfClient.ConvertPdfByteArrayToPdfa(pdfOriginalFile.ReadAllBytes(), false);
            file1.WriteAllBytes(response.PdfaByteArray);
            var response2 = webApiPdfClient.ConvertPdfByteArrayToPdfa(pdfOriginalFile.ReadAllBytes(), false);
            file2.WriteAllBytes(response2.PdfaByteArray);

There are two windows event logs created by the server when it hard crashes First Error Event Log Entry

Faulting application name: xxxxxxxxService.exe, version: 1.0.0.0, time stamp: 0x94123e39
Faulting module name: ucrtbase.dll, version: 10.0.17134.677, time stamp: 0x9f346d3f
Exception code: 0xc0000409
Fault offset: 0x000000000006e14e
Faulting process ID: 0x4f7c
Faulting application start time: 0x01d537b0b4116e7c
Faulting application path: C:\xxxxxxxxService.exe
Faulting module path: C:\WINDOWS\System32\ucrtbase.dll
Report ID: a32399dd-a1d3-4493-9f6c-ef6cb45453e4
Faulting package full name: 
Faulting package-relative application ID: 

Second Information Event Log Entry

Fault bucket 1930686170833522916, type 5
Event Name: BEX64
Response: Not available
Cab Id: 0

Problem signature:
P1: xxxxxxxxService.exe
P2: 1.0.0.0
P3: 94123e39
P4: ucrtbase.dll
P5: 10.0.17134.677
P6: 9f346d3f
P7: 000000000006e14e
P8: c0000409
P9: 0000000000000007
P10: 

Attached files:
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1B21.tmp.mdmp
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1D93.tmp.WERInternalMetadata.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DA4.tmp.xml
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DB2.tmp.csv
\\?\C:\ProgramData\Microsoft\Windows\WER\Temp\WER1DD2.tmp.txt

These files may be available here:
C:\ProgramData\Microsoft\Windows\WER\ReportArchive\AppCrash_xxxxxxxx_c7be53f830b5ebee58beaeda68b8ea707ef39a54_6dd26e0f_1fb623ad

Analysis symbol: 
Rechecking for solution: 0
Report Id: a32399dd-a1d3-4493-9f6c-ef6cb45453e4
Report Status: 268435456
Hashed bucket: fd129a9a61ef2bd3facb2cd7d94a90e4
Cab Guid: 0

I am just not sure what the cause of the hard crash is.

Answer 1

The problem was caused by a third party library I was not disposing as I should have done. The trial version of the library did not cause the issue but the licenced version did. Being semi intermittent and hard crashing made it much harder to locate. But as I narrowed it down it seemed like the garbage collector was having the clash with it.