.net azure azure-aks kestrel asp.net-core-8

Upgraded a Containerized .NET App From 6 to 8 and Startup Probe begins to fail in Azure Kubernetes Services

Recently, we've upgraded all our services to .NET 8 from .NET 6 and they all completely stopped working on our Azure Kubernetes Cluster. All our services fail by the health check in AKS not being able to reach the container, Hence, having AKS restart the container over and over.

Startup probe failed: Get "http://x.x.x.x:80/health/startup": dial tcp x.x.x.x:80: connect: connection refused

My first suspicion was the breaking change that happened to the Default Ports on .NET 8 Containerized Applications, that changed default port 80 to 8080, specified in the documentation, it says to set the environment variables ASPNETCORE_URLS, ASPNETCORE_HTTP_PORTS, ASPNETCORE_HTTPS_PORTS to the ports that you want the app to use.

It turns out that we already had all our services specify the ASPNETCORE_URLS in the environment variables of the container AND the appsettings.json file before we did the upgrade, which means that the application should be still using those ports. I also modified the Dockerfile to include those Environment variables just in case, but still not luck.

I tried increasing the timeout for the startup Probe and nothing happened either.

startupProbe:
 httpGet:
   path: /health/startup
   port: http
 timeoutSeconds: 30

Notes:

I have not changed ANY of the deployment.yaml, service.yaml, or ingress.yaml files for this app because they previously were listening and fully working on PORT 80.
When the app is downgraded to .NET 6 it works flawlessly without any other changes needed.

Dockerfile:

FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443
ENV ASPNETCORE_URLS=http://+:80
ENV ASPNETCORE_HTTP_PORTS=80
ENV ASPNETCORE_HTTPS_PORTS=443

FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY ["nuget.config", "."]
COPY ["NID.Support.API/NID.Support.API.csproj", "NID.Support.API/"]
RUN dotnet restore "NID.Support.API/NID.Support.API.csproj"
COPY . .
WORKDIR "/src/NID.Support.API"
RUN dotnet build "NID.Support.API.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "NID.Support.API.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "NID.Support.API.dll"]

EDIT:

It seems as if Kestrel service tries to start AFTER the application has been shutdown, not before. Can't tell why that is.

Logs from kubectl

[15:34:51 DBG] Registered model binder providers, in the following order: ["Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BinderTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ServicesModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.HeaderModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FloatingPointTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.EnumTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DateTimeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.SimpleTypeModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.TryParseModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CancellationTokenModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ByteArrayModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormFileModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.FormCollectionModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.KeyValuePairModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.DictionaryModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ArrayModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.CollectionModelBinderProvider", "Microsoft.AspNetCore.Mvc.ModelBinding.Binders.ComplexObjectModelBinderProvider"]
[15:34:53 DBG] Hosting starting
[15:34:53 WRN] Storing keys in a directory '/root/.aspnet/DataProtection-Keys' that may not be persisted outside of the container. Protected data will be unavailable when container is destroyed. For more information go to https://aka.ms/aspnet/dataprotectionwarning
[15:34:53 INF] User profile is available. Using '/root/.aspnet/DataProtection-Keys' as key repository; keys will not be encrypted at rest.
[15:34:53 DBG] Repository contains no viable default key. Caller should generate a key with immediate activation.
[15:34:53 DBG] Policy resolution states that a new key should be added to the key ring.
[15:34:53 INF] Creating key {xxxx} with creation date 2024-06-12 15:34:53Z, activation date 2024-06-12 15:34:53Z, and expiration date 2024-09-10 15:34:53Z.
[15:34:53 DBG] Descriptor deserializer type for key {xxxx} is 'Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.ConfigurationModel.AuthenticatedEncryptorDescriptorDeserializer, Microsoft.AspNetCore.DataProtection, Version=9.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'.
[15:34:53 DBG] No key escrow sink found. Not writing key {xxxx} to escrow.
[15:34:53 WRN] No XML encryptor configured. Key {xxxx} may be persisted to storage in unencrypted form.
[15:34:53 INF] Writing data to file '/root/.aspnet/DataProtection-Keys/key-xxxx.xml'.
[15:34:53 DBG] Key cache expiration token triggered by 'CreateNewKey' operation.
[15:34:53 DBG] Reading data from file '/root/.aspnet/DataProtection-Keys/key-xxxx.xml'.
[15:34:53 DBG] Found key {xxxx}.
[15:34:53 DBG] Considering key {xxxx} with expiration date 2024-09-10 15:34:53Z as default key.
[15:34:53 DBG] Using managed symmetric algorithm 'System.Security.Cryptography.Aes'.
[15:34:53 DBG] Using managed keyed hash algorithm 'System.Security.Cryptography.HMACSHA256'.
[15:34:53 DBG] Using key {xxxx} as the default key.
[15:34:53 DBG] Key ring with default key {xxxx} was loaded during application startup.
[15:36:05 INF] Application is shutting down...
[15:36:05 WRN] Overriding HTTP_PORTS '80' and HTTPS_PORTS '443'. Binding to values defined by URLS instead 'http://*:80;'.
[15:36:06 DBG] Middleware configuration started with options: {AllowedHosts = *, AllowEmptyHosts = True, IncludeFailureMessage = True}
[15:36:06 DBG] Wildcard detected, all requests with hosts will be allowed.
[15:36:06 ERR] Hosting failed to start
System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)

System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.BindAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Server.Kestrel.Core.KestrelServerImpl.StartAsync[TContext](IHttpApplication`1 application, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Hosting.GenericWebHostService.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__14_1(IHostedService service, CancellationToken token)
   at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)
   at Microsoft.Extensions.Hosting.Internal.Host.StartAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.Run(IHost host)
   at Program.<Main>$(String[] args) in /src/NID.Support.API/Program.cs:line 182

Solution

For anyone experiencing the same situation. Here was the solution to my problems. There was a NuGet Package causing the issues, in their GitHub page there was no release notes at the time for the new version so it would've been almost impossible to know apart from just updating everything we could for our project.

We upgraded and the Web API started working again without any issues.

<PackageReference Include="Microsoft.ApplicationInsights.Kubernetes" Version="6.1.2" />

<PackageReference Include="Microsoft.ApplicationInsights.Kubernetes" Version="7.0.0" />