I have just installed an EFK stack on my Kubernetes cluster using the guide on https://medium.com/@timfpark/efk-logging-on-kubernetes-on-azure-4c54402459c4
I have it working when accessing it through the proxy as stated in the guide on
http://localhost:8001/api/v1/namespaces/kube-system/services/kibana-logging/proxy
However, I want it to work through my existing ingress controller so I have created a new ingress rule using the yaml below:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
generation: 2
labels:
app: kibana
name: kibana
namespace: kube-system
spec:
rules:
- host: kibana.dev.example1.com
http:
paths:
- backend:
serviceName: kibana-logging
servicePort: 5601
path: /
status:
loadBalancer:
ingress:
- {}
To my service which runs as:
apiVersion: v1
kind: Service
metadata:
labels:
addonmanager.kubernetes.io/mode: Reconcile
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
kubernetes.io/name: Kibana
name: kibana-logging
namespace: kube-system
spec:
clusterIP: X.X.195.49
ports:
- port: 5601
protocol: TCP
targetPort: ui
selector:
k8s-app: kibana-logging
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
However, when I try and access my URL:
http://kibana.dev.example1.com
I get: {"statusCode":404,"error":"Not Found","message":"Not Found"}
If I try and access: http://kibana.dev.example1.com/app/kibana#
I get: "Kibana did not load properly. Check the server output for more information."
After looking through the logs for both Kibana pod and ingress pod and comparing the results between a successful request through the proxy and an unsuccessful request through the ingress I can see that...
for hitting /
"GET / HTTP/1.1" 200 197 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 491 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 197 0.003 200 6101a7003003d34636d2012e53c23ca7
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/app/kibana HTTP/1.1" 404 85 "http://kibana.dev.example1.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 612 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 5809ac2b33d3e23b200b13c9971d8520
for hitting /app/kibana#
"GET /app HTTP/1.1" 404 85 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 470 0.003 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.003 404 54a4abe0cae6d3d4298847a0db0786d6
"GET /app/kibana HTTP/1.1" 200 13301 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 477 0.041 [kube-system-kibana-logging-5601] X.X.22.204:5601 13272 0.040 200 6cb7e7698f5c72e0cd06b3408d8d4673
"GET /api/v1/namespaces/kube-system/services/kibana-logging/proxy/bundles/kibana.style.css?v=16627 HTTP/1.1" 404 85 "https://kibana.dev.example1.com/app/kibana" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15" 576 0.004 [kube-system-kibana-logging-5601] X.X.22.204:5601 85 0.004 404 0b825f03c36c2225ab082c2a0bab15f4
When hitting through the proxy most of these requests return 302s rather than 404s. Can the ingress not access these URLs in namespace kube-system?
Am I missing something obvious here - maybe hitting the wrong Kibana URL? I've done a lot of googling and can't find anything similar.
Turns out the problem was with the kibana configuration.
In the kibana deployment yaml there is an environment variable called SERVER_BASEPATH which is set to point at the kibana service proxy. This was causing the URL to be rewritten each time I tried to access the endpoint externally.
If you comment out this variable and it's value and redeploy kibana then it should work by just hitting the ingress address.