Search code examples
azureazure-service-fabricazure-service-fabric-mesh

How to fix Service Fabric Mesh failing due to docker errors


I've been working with Service Fabric Mesh Preview to test a service. Everything was working fine and I was even able to publish to Azure, then after I tried to add an additional service mesh app to the solution, everything fails, my local cluster does not switch to Mesh Nodes anymore due to errors generated by docker. The log is below, from what I saw it fails trying to create a network, and says object already exists, though, when I list docker ntworks from command line, it only shows the default ones, but not the one in the logs: "servicefabric_nat" I already tried uninstalling everything (Docker, SDKs, VS Tools, removing registry keys) and reinstlaling, still same issue. What I should to to fix the error?

Thanks

**********************
Windows PowerShell transcript start
Start time: 20190331163515
Username: AzureAD\EduardoFonseca
RunAs User: AzureAD\EduardoFonseca
Configuration Name: 
Machine: DESKTOP-CAP3E26 (Microsoft Windows NT 10.0.17134.0)
Host Application: PowerShell.exe -WindowStyle Hidden -NonInteractive -ExecutionPolicy RemoteSigned -Command & 'C:\Program Files\Microsoft SDKs\Service Fabric\ClusterSetup\DevClusterSetup.ps1' -Auto -PathToClusterLogRoot C:\SFDevCluster\Log -SetupLogFileName DevClusterSetup.log -CreateOneNodeCluster -CreateMeshCluster
Process ID: 12248
PSVersion: 5.1.17134.407
PSEdition: Desktop
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1.17134.407
BuildVersion: 10.0.17134.407
CLRVersion: 4.0.30319.42000
WSManStackVersion: 3.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: 1.1.0.1
**********************
Transcript started, output file is C:\SFDevCluster\Log\DevClusterSetup.log
Installing certificates for secure cluster setup...
Cleaning existing certificates...
Certificates removed.
Installing new certificates...
WARNING: This will install certificate with 'CN=ServiceFabricDevClusterCert' in following stores:

    # LocalMachine\My
    # LocalMachine\root &
    # CurrentUser\My

The CleanCluster.ps1 will clean these certificates or you can clean them up using script 'CertSetup.ps1 -Clean -CertSubjectName CN=ServiceFabricDevClusterCert'.
FabricSetup.exe invoked with arguments (C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\FabricSetup.exe
 /operation:installsfvolumedriver). Current Exe version 6.4.644.9590
Environment information Data Root C:\ProgramData\Microsoft Service Fabric, Log Root C:\ProgramData\Microsoft Service Fab
ric\log
Starting service eventlog
Starting service pla
Starting DriverInstallManager::Install
SFBDMiniport driver install path : C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\SFBDMiniport.inf for
 uninstalling device.
SFBDMiniport driver uninstalled sucessfully.
Sign Location is empty ''
Installing SFBDMiniport Driver
SFBDMiniport driver path : C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\SFBDMiniport.inf for install
ing device.
SFBDMiniport installed from path : C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code\SFBDMiniport.inf.
DriverInstallManager::Install successful
Operation successful
Creating Docker network and adding firewall rule...

Status   Name               DisplayName
------   ----               -----------
Running  docker             Docker Engine
docker : Error: No such network: servicefabric_nat
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error: No such ...rvicefabric_nat:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
docker : Error response from daemon: HNS failed with error : The object already exists.
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:590 char:35
+ ... onContent = docker $dockercommandParameters.Split(" ") 2> $tempErrorF ...
+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Error response ...already exists.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
WARNING: Unable to communicate with Docker. Validate the following
WARNING:     Docker is installed on the machine
WARNING:     Docker service is running
WARNING:     Docker is configured to run Windows containers
WARNING:     Open a command prompt and make sure the following command succeeds
WARNING:         docker network create -d=nat --subnet="10.128.0.0/24" --gateway="10.128.0.1" servicefabric_nat
DockerNetworkCommandFailed : Unable to communicate with Docker. Visit FAQ at https://aka.ms/sfmesh for details
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:617 char:9
+         DockerNetworkCommandFailed $warningOnFail
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,DockerNetworkCommandFailed
DockerNetworkCommandFailed : Unable to communicate with Docker. Visit FAQ at https://aka.ms/sfmesh for details
At C:\Program Files\Microsoft SDKs\Service Fabric\Tools\Scripts\ClusterSetupUtilities.psm1:617 char:9
+         DockerNetworkCommandFailed $warningOnFail
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Write-Error], WriteErrorException
    + FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,DockerNetworkCommandFailed

PS>TerminatingError(): "ScriptHalted"
>> TerminatingError(): "ScriptHalted"
>> TerminatingError(): "ScriptHalted"
>> TerminatingError(): "ScriptHalted"
>> TerminatingError(): "ScriptHalted"
ScriptHalted
**********************
Windows PowerShell transcript end
End time: 20190331163742
**********************

Solution

  • Already found the cause of the issue, apparently docker could not create the network because the network adapter already existed, they were listed as Network Adapters in Windows, here is how I solved the issue: * Uninstall Hyper-V * Uninstall the Hyper-V networks using Device Manager * Reinstall Hyper-V

    With that Service Fabric Local cluster started to work as expected again.