Service Fabric Explorer Health State Unknown

Question

The node in my partition keeps switching between Health State = OK and Health State = unknown. Sometimes the node disappears.

I have tried deleting the service, the app and unprovisioning the type, then redeploying, however I get the same problem.

It is a Service Fabric stateful service, and it's running fine locally, the issue I'm having is only in my dev environment.

I'm using 5 nodes.

ServiceManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="Integration.Optical.ServicePkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <!-- This is the name of your ServiceType. 
         This name must match the string used in the RegisterServiceAsync call in Program.cs. -->
    <StatefulServiceType ServiceTypeName="Integration.Optical.ServiceType" />
  </ServiceTypes>

  <!-- Code package is your service executable. -->
  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ExeHost>
        <Program>Integration.Optical.Service.exe</Program>
        <WorkingFolder>CodePackage</WorkingFolder>
      </ExeHost>
    </EntryPoint>
    <EnvironmentVariables>
      <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value=""/>
      <EnvironmentVariable Name="KEYVAULT_ENDPOINT" Value=""/>
    </EnvironmentVariables>
  </CodePackage>

  <!-- Config package is the contents of the Config directoy under PackageRoot that contains an 
       independently-updateable and versioned set of custom configuration settings for your service. -->
  <ConfigPackage Name="Config" Version="1.0.0" />

  <Resources>
    <Endpoints>
      <!-- This endpoint is used by the communication listener to obtain the port on which to 
           listen. Please note that if your service is partitioned, this port is shared with 
           replicas of different partitions that are placed in your code. -->
      <Endpoint Name="ServiceEndpoint" />

      <!-- This endpoint is used by the replicator for replicating the state of your service.
           This endpoint is configured through a ReplicatorSettings config section in the Settings.xml
           file under the ConfigPackage. -->
      <Endpoint Name="ReplicatorEndpoint" />
    </Endpoints>
  </Resources>
</ServiceManifest>

ApplicationManifest.xml:

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" ApplicationTypeName="Integration.OpticalType" ApplicationTypeVersion="1.0.0" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="Integration.Optical.Service_ASPNETCORE_ENVIRONMENT" DefaultValue="" />
    <Parameter Name="Integration.Optical.Service_KEYVAULT_ENDPOINT" DefaultValue="" />
    <Parameter Name="Integration.Optical.Service_MinReplicaSetSize" DefaultValue="3" />
    <Parameter Name="Integration.Optical.Service_PartitionCount" DefaultValue="1" />
    <Parameter Name="Integration.Optical.Service_TargetReplicaSetSize" DefaultValue="3" />
  </Parameters>
  <!-- Import the ServiceManifest from the ServicePackage. The ServiceManifestName and ServiceManifestVersion 
       should match the Name and Version attributes of the ServiceManifest element defined in the 
       ServiceManifest.xml file. -->
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="Integration.Optical.ServicePkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides />
    <EnvironmentOverrides CodePackageRef="code">
      <EnvironmentVariable Name="ASPNETCORE_ENVIRONMENT" Value="[Integration.Optical.Service_ASPNETCORE_ENVIRONMENT]" />
      <EnvironmentVariable Name="KEYVAULT_ENDPOINT" Value="[Integration.Optical.Service_KEYVAULT_ENDPOINT]" />
    </EnvironmentOverrides>
  </ServiceManifestImport>
  <DefaultServices>
    <!-- The section below creates instances of service types, when an instance of this 
         application type is created. You can also create one or more instances of service type using the 
         ServiceFabric PowerShell module.

         The attribute ServiceTypeName below must match the name defined in the imported ServiceManifest.xml file. -->
    <Service Name="Integration.Optical.Service" ServicePackageActivationMode="ExclusiveProcess">
      <StatefulService ServiceTypeName="Integration.Optical.ServiceType" TargetReplicaSetSize="[Integration.Optical.Service_TargetReplicaSetSize]" MinReplicaSetSize="[Integration.Optical.Service_MinReplicaSetSize]">
        <UniformInt64Partition PartitionCount="[Integration.Optical.Service_PartitionCount]" LowKey="-9223372036854775808" HighKey="9223372036854775807" />
      </StatefulService>
    </Service>
  </DefaultServices>
</ApplicationManifest>

ApplicationParameters/Cloud.xml:

<?xml version="1.0" encoding="utf-8"?>
<Application xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" Name="fabric:/Integration.Optical" xmlns="http://schemas.microsoft.com/2011/01/fabric">
  <Parameters>
    <Parameter Name="Integration.Optical.Service_ASPNETCORE_ENVIRONMENT" Value="" />
    <Parameter Name="Integration.Optical.Service_KEYVAULT_ENDPOINT" Value="" />
    <Parameter Name="Integration.Optical.Service_PartitionCount" Value="1" />
    <Parameter Name="Integration.Optical.Service_MinReplicaSetSize" Value="1" />
    <Parameter Name="Integration.Optical.Service_TargetReplicaSetSize" Value="1" />
  </Parameters>
</Application>

Answer 1

Not sure what part of this fixed it. But this is what I did and it's now working:

In ServiceManifest.xml I added HasPersistedState = true:

<StatefulServiceType ServiceTypeName="Integration.Optical.ServiceType" HasPersistedState="true" />

I moved the app configuration code

ServiceRuntime.RegisterServiceAsync...

from Service.RunAsync() to Program.Main()