Search code examples
c#asp.net-web-apiasp.net-coreazure-service-fabricservice-fabric-stateless

AspNet Core WebApi fails at startup with error System.Collections.Generic.KeyNotFoundException


I had my build successfully built + deployed in an Azure cluster. VSTS Build Tasks

I have no code changes. I compared working and not working build artifacts. I have new entries in my MyCompany.MyService.Api.deps.json file:

    ...
    "compilationOptions": {
        "defines": [
    ...
          "NETFRAMEWORK",
    ...
        ],
    ...
              "Microsoft.Bcl.Build": "1.0.21",
              "System.IdentityModel": "4.0.0.0",
              "System.ServiceModel.Web": "4.0.0.0",
              "System.Web.Services": "4.0.0.0",
              "System.Runtime.Caching": "4.0.0.0",
              "System.ServiceModel.Activation": "4.0.0.0"
            },
    ...
              "System.Transactions": "4.0.0.0",
              "System.Web": "4.0.0.0",
              "System.Diagnostics.Tracing.Reference1": "4.0.20.0",
              "System.Runtime.Serialization.Primitives.Reference": "4.0.10.0",
              "System.Runtime.Serialization.Xml.Reference": "4.0.10.0"
            },
    ...
          "System.Runtime.Reference1": "4.0.20.0",
          "System.Threading.Tasks.Reference1": "4.0.10.0",
          "System.Resources.ResourceManager.Reference1": "4.0.0.0",
          "System.Globalization.Reference1": "4.0.10.0",
          "System.Diagnostics.Tools.Reference1": "4.0.0.0",
          "System.Diagnostics.Debug.Reference1": "4.0.10.0",
          "System.Linq.Reference1": "4.0.0.0",
          "System.Collections.Reference1": "4.0.10.0",
          "System.Threading.Reference1": "4.0.10.0",
          "System.Runtime.Extensions.Reference1": "4.0.10.0",
          "System.IO.Reference1": "4.0.10.0",
          "System.Runtime.InteropServices.Reference1": "4.0.20.0",
          "System.Text.Encoding.Reference1": "4.0.10.0",
          "System.Reflection.Primitives.Reference1": "4.0.0.0",
          "System.Reflection.Reference1": "4.0.10.0",
          "System.Text.Encoding.Extensions.Reference1": "4.0.10.0",
          "System.ComponentModel.Reference": "4.0.0.0",
          "System.Reflection.Extensions.Reference1": "4.0.0.0"
...

I was successfully building and deploying for the past several months. Yet for the last 5 days, code that is built from the VSTS build is unable to deploy to the cluster. My API service fails to start, and I get the below error. Please suggest how I can debug. Code successfully deploys and launches in my local cluster.

Unhealthy event: SourceId='System.RA', Property='ReplicaOpenStatus', HealthState='Warning', ConsiderWarningAsError=false.
Replica had multiple failures during open on _ntsfmain_0. API call: IStatelessServiceInstance.Open(); Error = System.Collections.Generic.KeyNotFoundException (-2146232969)
The given key was not present in the dictionary.
at System.ThrowHelper.ThrowKeyNotFoundException()
at System.Collections.Generic.Dictionary`2.get_Item(TKey key)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.CandidateResolver.ComputeClassification(String dependency)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.CandidateResolver.ComputeClassification(String dependency)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.CandidateResolver.ComputeClassification(String dependency)
at Microsoft.AspNetCore.Mvc.Internal.DefaultAssemblyPartDiscoveryProvider.CandidateResolver.d__4.MoveNext()
at System.Linq.Enumerable.d__17`2.MoveNext()
at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.GetApplicationPartManager(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcCoreServiceCollectionExtensions.AddMvcCore(IServiceCollection services)
at Microsoft.Extensions.DependencyInjection.MvcServiceCollectionExtensions.AddMvc(IServiceCollection services)
at MyCompany.MyService.Api.Startup.ConfigureDevelopmentServices(IServiceCollection services) in C:\agent_C\_work\19\s\Source\MyCompany.MyService.Api\Startup.cs:line 39
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.AspNetCore.Hosting.ConventionBasedStartup.ConfigureServices(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.Internal.WebHost.EnsureApplicationServices()
at Microsoft.AspNetCore.Hosting.Internal.WebHost.BuildApplication()
at Microsoft.AspNetCore.Hosting.WebHostBuilder.Build()
at MyCompany.MyService.Api.Api.<>c__DisplayClass4_0.b__1(String url, AspNetCoreCommunicationListener listener) in C:\agent_C\_work\19\s\Source\MyCompany.MyService.Api\Api.cs:line 91
at Microsoft.ServiceFabric.Services.Communication.AspNetCore.AspNetCoreCommunicationListener.OpenAsync(CancellationToken cancellationToken)
at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.d__19.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.ServiceFabric.Services.Runtime.StatelessServiceInstanceAdapter.d__13.MoveNext()
For more information see: http://aka.ms/sfhealth

Solution

  • This is a known bug related to asp.net core 2.1. We had the same issue a few weeks ago.

    There are some conflicting dependencies on installation of core 2.1 that affect other dot net versions. Is likely that VSTS build environment or SF got upgraded to core 2.1 and affected your old builds.

    You can:

    Downgrade to 2.0: In our case, we had some services targeting core 2.1 and some 1.1 and 2.0, We downgraded all services to core 2.0 and the problem got solved.

    Or,

    You can fix dependencies issue adding the following code before the line AddMvc() of you api.

    Will look like this:

    var manager = new ApplicationPartManager();
    manager.ApplicationParts.Add(new AssemblyPart(typeof(Startup).Assembly));
    manager.ApplicationParts.Add(new AssemblyPart(typeof(Project1.Project1Type).Assembly));
    
    services.AddSingleton(manager);
    services.AddMvc();
    

    Or,

    Add a global.json file to your solution.

    Like this:

    {
      "sdk": {
        "version": "2.1.200"
      }
    }
    

    Please follow this GitHub issues for more information: