Search code examples
c#azureauthenticationasp.net-core-webapiidentitymodel

Microsoft authentication middleware suddenly extremely slow in development environment


I have an ASP.NET Core Web API I'm developing locally. It uses two different authentication schemes, both JWT bearer schemes - one is for "user" tokens (issued by my own Identity provider) and the other is for "service" tokens which are issued by my Azure tenant. I'll add some code below to show the setup of these schemes, but I'm not doing anything unusual.

Just recently I'm finding that my protected endpoints are suddenly extremely slow to respond in my local development environment - in the order of 60-second response times. I have found this is only true for endpoints that require service tokens. For example, an endpoint that accepts user or service tokens and is called with a user token responds fine when the service token requirement is removed.

Also, if I put a breakpoint in some middleware that runs before authentication then this breakpoint is hit fast, then a huge delay before the breakpoint in the controller itself is hit.

So I've concluded that something inside the JWT bearer authentication middleware is causing the app to hang. But I have no idea what this might be. I've tried using Fiddler to inspect the internal network traffic being made by my app (netsh winhttp set proxy 127.0.0.1:8888) but nothing there that's particularly long-running. Below is the trace. One thing that stands out here is the absence of any request to the Azure tenant well-known discovery doc. The first two requests are to my own discovery doc for the user tokens. Everything else seems to be App Insights-related.

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
12  200 HTTPS   my.identity.local:44119 /.well-known/openid-configuration   2,104       application/json; charset=UTF-8 identity.app:800            
13  200 HTTPS   my.identity.local:44119 /.well-known/openid-configuration/jwks  463     application/json; charset=UTF-8 identity.app:800            
17  200 HTTP    www.msftconnecttest.com /connecttest.txt    22  max-age=30, must-revalidate text/plain  svchost:3132            
18  502 HTTP    ipv6.msftconnecttest.com    /connecttest.txt    512 no-cache, must-revalidate   text/html; charset=UTF-8    svchost:3132            
24  200 HTTPS   fe2cr.update.microsoft.com  /v6/ClientWebService/client.asmx    2,155   private text/xml; charset=utf-8 svchost:8068            
34  502 HTTP    ipv6.msftconnecttest.com    /connecttest.txt    512 no-cache, must-revalidate   text/html; charset=UTF-8    svchost:3132            
35  200 HTTP    www.msftconnecttest.com /connecttest.txt    22  max-age=30, must-revalidate text/plain  svchost:3132            
47  200 HTTPS   fe2cr.update.microsoft.com  /v6/ClientWebService/client.asmx    2,149   private text/xml; charset=utf-8 svchost:8068                    

I've also tried tapping into the JWT bearer events. The only two being hit for the Service scheme are MessageReceived (hit very fast) and AuthenticationFailed (after the long delay and as expected because the token being used isn't a service token) so no clues there.

Hard to believe such a huge delay is CPU-bound, but I really don't know what else I can do to investigate. Have any of you encountered this before? Any tips would be amazing because it's so frustrating having such an insanely slow development environment. Like I say, this has only started in the last couple of days and I've barely touched the API codebase. Is it possible I'm being subjected to some sort of Azure rate limiting? I have a paid Azure account. As promised, here's the extension method I use to set up this authentication scheme:

private static void AddServiceTokenAuthenticationIfRequired(this AuthenticationBuilder authBuilder, IServiceCollection services, AuthenticationSettings settings)
{
    if (!settings.SupportServiceTokens)
    {
        return;
    }

    services.AddSingleton<CustomServiceAuthenticationEvents>();

    var tenantId = settings.TenantId ?? throw new Exception("Token tenant not configured");
    var authority = $"https://sts.windows.net/{tenantId}";
    var audience = settings.DefaultTokenAudience ?? throw new Exception("Token audience not configured");

    authBuilder.AddJwtBearer(AuthenticationSchemes.Service, options =>
    {
        options.Audience = audience;
        options.Authority = new Uri(authority).AbsoluteUri;
        options.TokenValidationParameters.NameClaimType = JwtClaimTypes.Subject;
        options.TokenValidationParameters.ValidateIssuer = true;
        options.TokenValidationParameters.ValidIssuers = [$"{authority}/"];
        options.TokenValidationParameters.ValidateAudience = true;
        options.TokenValidationParameters.ValidateIssuerSigningKey = true;
        options.TokenValidationParameters.ValidateLifetime = true;
        options.TokenValidationParameters.LogValidationExceptions = true;
        options.EventsType = typeof(CustomServiceAuthenticationEvents);
        options.MapInboundClaims = false;
    });
}

Solution

  • OK, after 24 hours of banging my head against a wall I discovered that the massive latency between my dev laptop and Microsoft was being caused by the Cloudflare WARP app I'd installed to supposedly speed up my internet. I can't explain it but this app has caused me other problems in the past with network flakiness so I'm just uninstalling it. Disabling/uninstalling fixes the issue which, by the way, applied to all communication between my machine and Azure resources so not just authentication supporting resources.