Intermittent System.ArgumentNullException: Value cannot be null. (Parameter 'array') in System.Private.CoreLib

Question

I am having difficulty tracking down the cause of this exception.

Calling Code

// reduced for brevity
endpoints.MapGet("/sse/workorder", async Task (HttpContext context, CancellationToken token) => {
    while (!token.IsCancellationRequested)
    {
        await context.SSESendEventAsync(new SSEEvent(), token);
        await Task.Delay(TimeSpan.FromSeconds(2), token);
    }
}

public static async Task SSESendEventAsync(this HttpContext ctx, SSEEvent e, CancellationToken token)
{
    ArgumentNullException.ThrowIfNull(ctx);
    ArgumentNullException.ThrowIfNull(e);

    if (!string.IsNullOrWhiteSpace(e.Id))
    {
        await ctx.Response.WriteAsync($"id: {e.Id}\n", token);   // <-exception occurs here.
    }
    ...
}

Also, if pertinent, this continually runs asynchronously elsewhere in the program...

try
{
    while (!token.IsCancellationRequested)
    {
        await ctx.Response.WriteAsync(": \n\n", token);

        await Task.Delay(TimeSpan.FromSeconds(10), token);
    }
}
catch (Exception ex) when (ex is OperationCanceledException) { }

The Exception

15:48:40:643    Exception thrown: 'System.ArgumentNullException' in System.Private.CoreLib.dll
15:48:40:643    Exception thrown: 'System.ArgumentNullException' in System.Private.CoreLib.dll
15:48:40:643    Microsoft.AspNetCore.Routing.EndpointMiddleware: Information: Executed endpoint 'HTTP: GET /sse/workorder'
15:48:40:643    Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Error: An unhandled exception has occurred while executing the request.
15:48:40:643    
15:48:40:643    System.ArgumentNullException: Value cannot be null. (Parameter 'array')
15:48:40:643       at System.Buffers.SharedArrayPool`1.Return(T[] array, Boolean clearArray)
15:48:40:643       at System.IO.Pipelines.StreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
15:48:40:643       at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
15:48:40:643       at System.Threading.Tasks.ValueTask`1.GetTaskForValueTaskSource(IValueTaskSource`1 t)
15:48:40:643    --- End of stack trace from previous location ---
15:48:40:643       at SSE.SSEHttpContextExtensions.SSESendEventAsync(HttpContext ctx, SSEEvent e, CancellationToken token) in C:\[redacted]\SSEHttpContextExtensions.cs:line 50
15:48:40:643       at SSE.Startup.<>c__DisplayClass3_0.<<Configure>b__5>d.MoveNext() in C:\[redacted]\Startup.cs:line 529
15:48:40:643    --- End of stack trace from previous location ---
15:48:40:643       at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
15:48:40:643       at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)
15:48:40:643    Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Warning: The response has already started, the error page middleware will not be executed.
15:48:40:643    Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Error: An unhandled exception has occurred while executing the request.
15:48:40:643    
15:48:40:643    System.ArgumentNullException: Value cannot be null. (Parameter 'array')
15:48:40:643       at System.Buffers.SharedArrayPool`1.Return(T[] array, Boolean clearArray)
15:48:40:643       at System.IO.Pipelines.StreamPipeWriter.FlushAsyncInternal(Boolean writeToStream, ReadOnlyMemory`1 data, CancellationToken cancellationToken)
15:48:40:643       at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
15:48:40:643       at System.Threading.Tasks.ValueTask`1.GetTaskForValueTaskSource(IValueTaskSource`1 t)
15:48:40:643    --- End of stack trace from previous location ---
15:48:40:643       at SSE.SSEHttpContextExtensions.SSESendEventAsync(HttpContext ctx, SSEEvent e, CancellationToken token) in C:\[redacted]\SSEHttpContextExtensions.cs:line 50
15:48:40:643       at SSE.Startup.<>c__DisplayClass3_0.<<Configure>b__5>d.MoveNext() in C:\[redacted]\Startup.cs:line 529
15:48:40:643    --- End of stack trace from previous location ---
15:48:40:643       at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|7_0(Endpoint endpoint, Task requestTask, ILogger logger)
15:48:40:643       at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)
15:48:40:643       at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)
15:48:40:643       at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
15:48:40:643       at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
15:48:40:643       at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddlewareImpl.Invoke(HttpContext context)
15:48:40:643    Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Warning: The response has already started, the error page middleware will not be executed.
15:48:40:643    Microsoft.AspNetCore.Server.IIS.Core.IISHttpServer: Error: Connection ID "18158513719837982798", Request ID "40000052-0005-fc00-b63f-84710c7967bb": An unhandled exception was thrown by the application.

This exception (along with ArgumentOutOfRangeException) occurs intermittently and infrequently here. Sometimes days, sometimes minutes.

I would appreciate any help in finding the root cause, or any tips on how to investigate it further.

Answer 1

Kudos to @Richard's comment for steering me towards the solution. The issue was, in fact, that writing to HttpResponse is not thread safe. One solution is to use a semaphore (since locks can't be used with async) whenever writing a response, to restrict writing to one thread at a time.

Aside: For our situation, storing the semaphore within HttpContext.Items was chosen since this class is static but we need the lock to be contextual.

public static async Task SSESendEventAsync(this HttpContext ctx, SSEEvent e, CancellationToken token)
{
    ArgumentNullException.ThrowIfNull(ctx);
    ArgumentNullException.ThrowIfNull(e);

    if (!string.IsNullOrWhiteSpace(e.Id))
    {
        var writingLock = GetLock(context);
        await writingLock.WaitAsync(token);

        try
        {
            await ctx.Response.WriteAsync($"id: {e.Id}\n", token);
        }
        finally
        {
            writingLock.Release();
        }
    }
    ...
}

/// Get a semaphore to use as a writing lock.
/// Store it within the HTTP context for reference later.
private static SemaphoreSlim GetLock(HttpContext context)
{
    ArgumentNullException.ThrowIfNull(context);

    return (SemaphoreSlim)(context.Items["httpContextWritingLock"] ??= new SemaphoreSlim(1, 1));
}