There was an error during CodePackage activation.Container failed to start for image

Question

I have created an image for ASP.net MVC app from 4.7.2-windowsservercore-ltsc2016 When I run the container locally using Docker all works as it should. I have a 5 node Fabric Cluster in Azure set up and when I try to deploy initially I get:

Error event: SourceId='System.FM', Property='State'.
Partition is below target replica or instance count.
fabric:/xx/yy -1 1 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  (Showing 0 out of 0 replicas. Total available replicas: 0)

For more information see: http://aka.ms/sfhealth

And when I look at the individual nodes I see:

Error event: SourceId='System.Hosting', Property='Download:1.0:1.0'.
There was an error during download

After a while the Health events in the nodes change and show:

The ServicePackage was activated successfully.

then

System.Hosting | CodePackageActivation:Code:EntryPoint

There was an error during CodePackage activation.Container failed to start for image:myrepo.azurecr.io/repo:1.0.0. failed to create endpoint sf-0-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx_xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx on network nat: HNS failed with error : The object already exists.

and then

There was an error during CodePackage activation.The service host terminated with exit code:7148

and then

The ServiceType was unregistered on the node since the Runtime or ApplicationHost closed.

My service manifest has a full image name that is store in Azure CR:

<ImageName>myrepo.azurecr.io/repo:1.0.0</ImageName>

Also in my application manifest I have the repository credentials:

<RepositoryCredentials AccountName="myrepo" Password="xxxxxxxxxxxxxxxxxxxxxxxx" PasswordEncrypted="false"/>

I am happy to provide more details/logs, please just let me know where to get them from.

EDIT:

Service manifest:

<?xml version="1.0" encoding="utf-8"?>
<ServiceManifest Name="mymvcPkg"
                 Version="1.0.0"
                 xmlns="http://schemas.microsoft.com/2011/01/fabric"
                 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <ServiceTypes>
    <StatelessServiceType ServiceTypeName="mymvcType" UseImplicitHost="true" />
  </ServiceTypes>
  <CodePackage Name="Code" Version="1.0.0">
    <EntryPoint>
      <ContainerHost>
        <ImageName>myrepo.azurecr.io/repo:1.0.0</ImageName>
      </ContainerHost>
    </EntryPoint>    
  </CodePackage>
  <ConfigPackage Name="Config" Version="1.0.0" />
  <Resources>
    <Endpoints>
      <Endpoint Name="mymvcTypeEndpoint" UriScheme="http" Port="80" Protocol="http" />
    </Endpoints>
  </Resources>
</ServiceManifest>

Application manifest:

<?xml version="1.0" encoding="utf-8"?>
<ApplicationManifest ApplicationTypeName="MvcType"
                     ApplicationTypeVersion="1.0.0"
                     xmlns="http://schemas.microsoft.com/2011/01/fabric"
                     xmlns:xsd="http://www.w3.org/2001/XMLSchema"
                     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Parameters>
    <Parameter Name="mymvc_InstanceCount" DefaultValue="-1" />
  </Parameters>  
  <ServiceManifestImport>
    <ServiceManifestRef ServiceManifestName="mymvcPkg" ServiceManifestVersion="1.0.0" />
    <ConfigOverrides />
    <Policies>
      <ContainerHostPolicies CodePackageRef="Code">
        <RepositoryCredentials AccountName="myrepo" Password="xxxxxxxxxxxxx" PasswordEncrypted="false"/>
        <PortBinding ContainerPort="80" EndpointRef="mymvcTypeEndpoint"/>
      </ContainerHostPolicies>
    </Policies>
  </ServiceManifestImport>
  <DefaultServices>    
    <Service Name="mymvc" ServicePackageActivationMode="ExclusiveProcess">
      <StatelessService ServiceTypeName="mymvcType" InstanceCount="[mymvc_InstanceCount]">
        <SingletonPartition />
      </StatelessService>
    </Service>
  </DefaultServices>
</ApplicationManifest>

Answer 1

The problem I had was with the instance type I have used for my VMs (Nodes) in the cluster. Azure support found error in some of their logs:

System.ComponentModel.Win32Exception (0x80004005): Not enough storage is available to process this command

Switching to more powerful instance solved the issue. Microsoft states:

The minimum supported use VM SKU is Standard D1 or Standard D1_V2 or equivalent with a minimum of 14 GB of local SSD

Despite the VMs I have been trying to configure having 20GB they did not work. Currently I am running tests on SKU A2 and seems to be working fine so far. Microsoft also stated:

• Partial core VM SKUs like Standard A0 are not supported for production workloads

• Standard A1 SKU is not supported for production workloads for performance reasons.

Considering the above, D1_V2 (3.5GB RAM, 50GB SSD) is probably the cheapest viable option.