Search code examples
c#azureazure-worker-rolesazure-cloud-servicesazure-application-insights

Why is Application Insights Preventing my Worker Role from Starting?

I have an Azure Cloud Service with a single worker role that will deploy, start, and run without fail every time, but I ran into some intermittent problems I wanted to diagnose with Application Insights. I followed directions from this article as well as here.

After installing the nugget packages and adding the one line of code, I ran the cloud service locally in both debug and release mode, and was able to see the AI info being reported to the Application Insights resource. So I packaged it up and uploaded the new configuration to my cloud service.

But then from the cloud the worker role will not actually start--it goes into a recycle death spiral, and in the "Diagnose and Solve Problems" blade I get a variation of the error below, always about "System.Threading.AsyncLocal'1'" and not being able to load the RoleEntryPoint. Following this article did not shed much light onto things, as the error pretty much tells me why it keeps recycling but not any clues on what to do about it.

Production - WebReportDownloader_IN_0: BusyRole Waiting for role to start... Failed to load role entrypoint. System.TypeLoadException: Could not load type 'System.Threading.AsyncLocal`1' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'. at Microsoft.ApplicationInsights.Extensibility.SdkInternalOperationsMonitor.Exit() at Microsoft.ApplicationInsights.Extensibility.Implementation.TelemetryConfigurationFactory.Initialize(TelemetryConfiguration configuration, TelemetryModules modules, String serializedConfiguration) at Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.get_Active() at WebReportDownloader..ctor() in C:\Users\User\Source\Repos\Program\WebReportDownloader\WebReportDownloader.cs:line 21 --- End of inner exception stack trace --- at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean& bNeedSecurityCheck) at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) at System.Activator.CreateInstance(Type type, Boolean nonPublic) at System.Activator.CreateInstance(Type type) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.GetRoleEntryPoint(Assembly entryPointAssembly) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.CreateRoleEntryPoint(RoleType roleTypeEnum) at Microsoft.WindowsAzure.ServiceRuntime.RoleEnvironment.InitializeRoleInternal(RoleType roleTypeEnum) [2017-09-14T15:28:21Z] Last exit time: [2017/09/14, 15:28:21.944]. Last exit code: 0.

Normally I can get enough of a hint from searching the exception and/or details, but this time I can't find anything that resembles my problem. All I can guess is that something small but critical is awry with my configuration, but I haven't been able to determine where I've gone off the rails based on the instructions I linked above.

My ServiceConfiguration.Cloud file...

    <?xml version="1.0" encoding="utf-8"?>
<ServiceConfiguration serviceName="WebReportDownloader" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceConfiguration" osFamily="4" osVersion="*" schemaVersion="2015-04.2.6">
  <Role name="WebReportDownloader">
    <Instances count="2" />
    <ConfigurationSettings>
      <Setting name="Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString" value="UseDevelopmentStorage=true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.Enabled" value="true" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountUsername" value="admin" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountEncryptedPassword" value="ENCRYPTEDVALUE" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteAccess.AccountExpiration" value="2018-09-01T23:59:59.0000000-04:00" />
      <Setting name="Microsoft.WindowsAzure.Plugins.RemoteForwarder.Enabled" value="true" />
      <Setting name="APPINSIGHTS_INSTRUMENTATIONKEY" value="KEYVALUE" />
    </ConfigurationSettings>
    <Certificates>
      <Certificate name="Microsoft.WindowsAzure.Plugins.RemoteAccess.PasswordEncryption" thumbprint="847EEDE0084CF57A5A774CAE9E700713726CC856" thumbprintAlgorithm="sha1" />
    </Certificates>
  </Role>
</ServiceConfiguration>

And my ApplicationInsights.config...

<?xml version="1.0" encoding="utf-8"?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
    <TelemetryInitializers>
        <Add Type="Microsoft.ApplicationInsights.DependencyCollector.HttpDependenciesParsingTelemetryInitializer, Microsoft.AI.DependencyCollector"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureWebAppRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.BuildInfoConfigComponentVersionTelemetryInitializer, Microsoft.AI.WindowsServer"/>
    </TelemetryInitializers>
    <TelemetryModules>
        <Add Type="Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector">
            <ExcludeComponentCorrelationHttpHeadersOnDomains>
                <!-- 
        Requests to the following hostnames will not be modified by adding correlation headers. 
        This is only applicable if Profiler is installed via either StatusMonitor or Azure Extension.
        Add entries here to exclude additional hostnames.
        NOTE: this configuration will be lost upon NuGet upgrade.
        -->
                <Add>core.windows.net</Add>
                <Add>core.chinacloudapi.cn</Add>
                <Add>core.cloudapi.de</Add>
                <Add>core.usgovcloudapi.net</Add>
                <Add>localhost</Add>
                <Add>127.0.0.1</Add>
            </ExcludeComponentCorrelationHttpHeadersOnDomains>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector">
            <!--
      Use the following syntax here to collect additional performance counters:

      <Counters>
        <Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Handle Count" ReportAs="Process handle count" />
        ...
      </Counters>

      PerformanceCounter must be either \CategoryName(InstanceName)\CounterName or \CategoryName\CounterName

      NOTE: performance counters configuration will be lost upon NuGet upgrade.

      The following placeholders are supported as InstanceName:
        ??APP_WIN32_PROC?? - instance name of the application process  for Win32 counters.
        ??APP_W3SVC_PROC?? - instance name of the application IIS worker process for IIS/ASP.NET counters.
        ??APP_CLR_PROC?? - instance name of the application CLR process for .NET counters.
      -->
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryModule, Microsoft.AI.PerfCounterCollector"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.DeveloperModeWithDebuggerAttachedTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnhandledExceptionTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnobservedExceptionTelemetryModule, Microsoft.AI.WindowsServer">
            <!--</Add>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.FirstChanceExceptionStatisticsTelemetryModule, Microsoft.AI.WindowsServer">-->
        </Add>
    </TelemetryModules>
    <TelemetryProcessors>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryProcessor, Microsoft.AI.PerfCounterCollector"/>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.AutocollectedMetricsExtractor, Microsoft.ApplicationInsights"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
            <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
            <ExcludedTypes>Event</ExcludedTypes>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
            <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
            <IncludedTypes>Event</IncludedTypes>
        </Add>
    </TelemetryProcessors>
    <TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel"/>
<!-- 
    Learn more about Application Insights configuration with ApplicationInsights.config here: 
    http://go.microsoft.com/fwlink/?LinkID=513840

    Note: If not present, please add <InstrumentationKey>Your Key</InstrumentationKey> to the top of this file.
  --></ApplicationInsights>

And my "OnStart()" method where I actually set the InstrumentationKey...

public override bool OnStart()
        {
            // Set the maximum number of concurrent connections
            ServicePointManager.DefaultConnectionLimit = 12;
            TelemetryConfiguration.Active.InstrumentationKey = RoleEnvironment.GetConfigurationSettingValue("APPINSIGHTS_INSTRUMENTATIONKEY");

            // For information on handling configuration changes
            // see the MSDN topic at https://go.microsoft.com/fwlink/?LinkId=166357.

            bool result = base.OnStart();

            Trace.TraceInformation("WebReportDownloader has been started");

            return result;
        }

I would appreciate any insight (ha!) or guidance anyone can provide...

Update:

Through further experimentation, I've determined that whenever I'm trying to call the Application Insights assembly directly (such as when creating a Telemetry Client or setting the ApplicationInsights Resource Key directly) or when the ApplicationInsight TraceListener is called because of a trace the exception below is called

Could not load type 'System.Threading.AsyncLocal`1' from assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089'.

If I do the run-time instrumentation, I'm able to get some basic traces and logs to my Insight resources, but attempting to get anything more still escapes me.

Additionally, I'm currently targeting .NET 4.6.1 (information I neglected to mention previously).

Solution

  • After much back and forth with Azure support, I was finally able to get a screen sharing session set up. The helpful gentleman zeroed in on the problem almost immediately: Because my worker role project targets .NET 4.6.1 the ServiceConfiguration files (both .Cloud.cscfg and .Local.cscfg) need to be targeting osFamily 5.

    The defaults generated by the VS17 wizard set it at osFamily=4 as seen below...

    .NET 4.5.2 configuration

    However, what I needed to make everything run properly was to be targeting osFamily=5 as seen below...

    .NET 4.6.1 configuration

    That one small change allowed the web role to deploy properly while instantiating a TelemetryClient.