Failed calling webhook "namespace.sidecar-injector.istio.io"

I have make my deployment work with istio ingressgateway before. I am not aware of any changes made in istio or k8s side.

When I tried to deploy, I see an error in replicaset side that's why it cannot create new pod.

Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.104.136.116:443: connect: no route to host

When I try to go inside api-server and ping 10.104.136.116 (istiod service IP) it just hangs.

What I have tried so far:

Deleted all coredns pods
Deleted all istiod pods
Deleted all weave pods
Reinstalling istio via istioctl x uninstall --purge
turning all of VMs firewall
sudo iptables -P INPUT ACCEPT sudo iptables -P FORWARD ACCEPT sudo iptables -P OUTPUT ACCEPT sudo iptables -F
restarted all of the nodes
manual istio pod injection

Setup

k8s version: 1.21.2
istio: 1.10.3
HA setup
CNI: weave
CRI: containerd

Solution

In my case it was due to firewall. Following this Istio debug guide, I identified that the kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4 command was timing out while all other cluster internal calls were ok.

The best way to diagnose this is to open temporarly your AWS Security Groups involved to 0.0.0.0/0 for port 15017 and then try again. If the errror won't show again, you know there's need to fix this part.

I am using EKS with Amazon VPC CNI v1.12.2-eksbuild.1