ASP.NET Core slow HTTP requests to Azure App Service

Question

I've got a ASP.NET Core middleware that calls another HTTP service to check if user is authorized to proceed with request or not. At the moment it depends on a provided custom header, called X-Parameter-Id.

using System;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.Configuration;

namespace ParameterAuthorization.Middleware
{
    public class ParameterAuthorizationMiddleware
    {
        private readonly RequestDelegate _next;
        private readonly IParameterAuthorizationService _parameterAuthorizationService;

        public ParameterAuthorizationMiddleware(RequestDelegate next, IParameterAuthorizationService parameterAuthorizationService)
        {
            _next = next ?? throw new ArgumentNullException(nameof(next));
            _parameterAuthorizationService = parameterAuthorizationService ?? throw new ArgumentNullException(nameof(parameterAuthorizationService));
        }

        public async Task InvokeAsync(HttpContext httpContext, IConfiguration configuration)
        {
            if (httpContext is null)
            {
                throw new ArgumentNullException(nameof(httpContext));
            }

            if (parameterRequestContext is null)
            {
                throw new ArgumentNullException(nameof(parameterRequestContext));
            }

            if (!(httpContext.Request.Headers.ContainsKey("X-Parameter-Id") && httpContext.Request.Headers.ContainsKey("Authorization")))
            {
                await ForbiddenResponseAsync(httpContext);
            }

            var parameterIdHeader = httpContext.Request.Headers["X-Parameter-Id"].ToString();

            if (!int.TryParse(parameterIdHeader, out var parameterId) || parameterId < 1)
            {
                await ForbiddenResponseAsync(httpContext);
            }

            var authorizationHeader = httpContext.Request.Headers["Authorization"].ToString();

            var parameterResponse = await _parameterAuthorizationService.AuthorizeUserParameterAsync(parameterId, authorizationHeader);

            if (string.IsNullOrWhiteSpace(parameterResponse))
            {
                await ForbiddenResponseAsync(httpContext);
            }

            await _next.Invoke(httpContext);
        }

        private static async Task ForbiddenResponseAsync(HttpContext httpContext)
        {
            httpContext.Response.StatusCode = StatusCodes.Status403Forbidden;
            await httpContext.Response.WriteAsync("Forbidden");
            return;
        }
    }
}

And that's the HTTP call implementation:

using System;
using System.IO;
using System.Net.Http;
using System.Threading.Tasks;
using Newtonsoft.Json;

namespace ParameterAuthorization.Middleware.Http
{
    public class ParameterAuthorizationService : IParameterAuthorizationService
    {
        private readonly HttpClient _httpClient;
        private readonly JsonSerializer _jsonSerializer;

        public ParameterAuthorizationService(HttpClient httpClient, JsonSerializer jsonSerializer)
        {
            _httpClient = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
            _jsonSerializer = jsonSerializer ?? throw new ArgumentNullException(nameof(jsonSerializer));
        }

        public async Task<string> AuthorizeUserParameterAsync(int parameterId, string authorizationHeader)
        {
            var request = CreateRequest(parameterId, authorizationHeader);

            var result = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

            if (!result.IsSuccessStatusCode)
            {
                return string.Empty;
            }

            using (var responseStream = await result.Content.ReadAsStreamAsync())
            using (var streamReader = new StreamReader(responseStream))
            using (var jsonTextReader = new JsonTextReader(streamReader))
            {
                return _jsonSerializer.Deserialize<ParameterResponse>(jsonTextReader).StringImInterestedIn;
            }
        }

        private static HttpRequestMessage CreateRequest(int parameterId, string authorizationHead1er)
        {
            var parameterUri = new Uri($"parameters/{parameterId}", UriKind.Relative);

            var message = new HttpRequestMessage(HttpMethod.Get, parameterUri);

            message.Headers.Add("Authorization", authorizationHead1er);

            return message;
        }
    }
}

And this is the boilerplate code to DI named HttpClient

sc.TryAddSingleton<JsonSerializer>();
sc.AddHttpClient<IParameterAuthorizationService, ParameterAuthorizationService>(client =>
{
    client.BaseAddress = authorizationServiceUri;
    client.DefaultRequestHeaders.Add("Accept", "application/json");
});

authorizationServiceUri is something I provide from a custom extension method.

The issue is that my calls to this service will randomly take 7, 10, even 20 seconds to this service and then it's going to be quick and then again slow. I call this exact ParameterAuthorizationService from Postman, it takes less than 50ms, constantly.

I'm attaching a screenshot from Application Insights showing the whole sequence of events.

Both services are deployed as Azure App Services under the same subscription within the same App Service Plan.

Code works just fine, but I'm already pulling my hair off having no clue what could be causing these performance abnormalities.

I've also checked TCP Connections in Azure App service and it's all green.

What could be the reason that some HTTP calls will be really slow?

Update

My App Service runs on a S1 App Service Plan. https://azure.microsoft.com/en-us/pricing/details/app-service/windows/

Answer 1

You should short circuit the pipeline if you write to httpContent. See Aspnet Core Middleware documentation :

Don't call next.Invoke after the response has been sent to the client.

Use something like this :

if (string.IsNullOrWhiteSpace(parameterResponse))
{
    await ForbiddenResponseAsync(httpContext);
}
else
{
    await _next.Invoke(httpContext);
}

Consider also handling authentification with a middleware that inherit from the aspnet core AuthenticationHandler class to leverage all the aspnet core Authentification/Authorization facilities. Here is an implementation example of a BasicAuthentification handler for simplicity.

Your ParameterAuthorizationService looks good. I don't think it's the source of your slow request call. To be sure, you can track the entire service call by measuring the time it takes and publishing-it in appinsights :

public class ParameterAuthorizationService : IParameterAuthorizationService
{
    //...

    private readonly TelemetryClient _telemetryClient;

    public ParameterAuthorizationService(HttpClient httpClient, JsonSerializer jsonSerializer)
    {
        //...

        _telemetryClient = new TelemetryClient();
    }

    public async Task<string> AuthorizeUserParameterAsync(int parameterId, string authorizationHeader)
    {
        var startTime = DateTime.UtcNow;
        var timer = Stopwatch.StartNew();
        var isSuccess = true;

        try
        {
            return await AuthorizeUserParameterImpl(parameterId, authorizationHeader);
        }
        catch (Exception ex)
        {
            timer.Stop();
            isSuccess = false;

            _telemetryClient.TrackException(ex);
            _telemetryClient.TrackDependency("Http", nameof(ParameterAuthorizationService), nameof(AuthorizeUserParameterAsync),
                startTime, timer.Elapsed, isSuccess);
            throw;
        }
        finally
        {
            if (timer.IsRunning)
                timer.Stop();
            if (isSuccess)
                _telemetryClient.TrackDependency(
                "Http", nameof(ParameterAuthorizationService), nameof(AuthorizeUserParameterAsync),
                startTime, timer.Elapsed, isSuccess);
        }                
    }

    private async Task<string> AuthorizeUserParameterImpl(int parameterId, string authorizationHeader)
    {
        //Your original code
    }
}