Facing issues in setting up Cadence Workflow (Cadence server) in K8

I am facing issues in setting up cadence in Kubernetes environment, following are details

CADENCE_SERVER_IMAGE_VERSION: 0.19.2
All Cadence components are running within single pod
It seems Ringpop configuration requires headless services, but headless doesnt works with Istio
Every things works fine if i just have one single Pod, but as soon as I create 2 pods they start fighting for tasklist and shards and thats what my problem is Ringpop config: { name: RINGPOP_SEEDS, value: 'api-gtp-cadence.api-gtp-cadence.svc.cluster.local:7933,api-gtp-cadence.api-gtp-cadence.svc.cluster.local:7934,api-gtp-cadence.api-gtp-cadence.svc.cluster.local:7935,api-gtp-cadence.api-gtp-cadence.svc.cluster.local:7939'

Solution

Every things works fine if i just have one single Pod, but as soon as I create 2 pods they start fighting for tasklist and shard

Headless is required for Cadence in K8s. This is because Cadence uses Ringpop to manage the membership of each node of the cluster. Ringpop uses IP:port of each node as identity and address to communicate. Headless is the only way(AFAIK) that will give each pod a PodIP. That's why only single pod cluster work without headless -- because at that case, there is only one member in the ringpop.