Search code examples
azure-service-fabricservice-fabric-stateful

How to resolve: warning on every partition after reducing TargetReplicaSetSize

I've got the following warning on Service Fabric cluster:

Replica had multiple failures during open on _NodeType1_2. API call: IStatefulServiceReplica.Open(); Error = System.ArgumentException (-2147024809)

That happens after reducing (MinimalReplicaSetSize, TargetReplicaSetSize) from (3,5) to (3, 3). I can see replica set was reduced but every replica which was removed reporting mentioned warning - in my case 2 replicas has warning.

Is there any way to fix this without deleting stateful service?

Cluster is running on 6.3.176.9494 SF runtime.

Solution

  • I was managed to solve this issue by killing stateful service process on nodes(one by one - to keep state) which reported warnings.

    I have noticed one more side effect: on every such nodes that service process utilized a lot of CPU(30-40%) - after killing and auto-restarting by SF it calms down.