I am currently encountering an issue with my Amazon EKS cluster that hosts Lucidworks Fusion. To start with I had an Amazon EKS v1.18 cluster that I then upgraded to v1.19 which all went well. I also had ingress-nginx-3.7.1 running in my cluster that I then upgraded to ingress-nginx-4.0.19 everything still seemed to operate properly at that point. That said once I upgraded my cluster to v1.20, when draining my pods and restarting them in my new worker nodes all pods restarted properly except my Ambassador pod which is stuck in a CrashLoopBackOff state now. I looked into this pod's logs and found the following error:
2022/04/08 14:00:35 ERROR(s): kubebootstrap: WORKER PANICKED: ingresses.networking.k8s.io is forbidden: User "system:serviceaccount:sandbox1:sandbox1-ambassador" cannot list resource "ingresses" in API group "networking.k8s.io" in the namespace "sandbox1"
Does anyone know what this means ? I believe this is telling me something related to permissions but I am not exactly sure. Also I would like to understand what caused this issue, was it upgrading my EKS cluster to v1.20 or was it upgrading ingress-nginx or something entirely different ? Lastly I'd like to understand how I can solve this issue and get this pod running properly again ? Any help is welcomed.
After digging into this further I found that in order to fix this issue I needed to edit the sandbox1-ambassador role. So I executed the following command
kubectl edit role sandbox1-ambassador
and I appended the following to this configuration:
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs:
- list
- watch
- get
Once I saved these changes and closed the edit session I restarted the Ambassador pod and everything started working properly again.