Search code examples
kubernetesopenshiftprometheusmonitoringgrafana

Prometheus datasource : client_error: client error: 403

Hi I am trying to add built-in OpenShift(v4.8) prometheus data source to a local grafana server. I have given basic auth with username and password and as of now I have enabled skip tls verify also. Still I'm getting this error

error

Prometheus URL = https://prometheus-k8s-openshift-monitoring.apps.xxxx.xxxx.xxxx.com

this is the grafana log


logger=tsdb.prometheus t=2022-04-12T17:35:23.47+0530 lvl=eror msg="Instant query failed" query=1+1 err="client_error: client error: 403"
logger=context t=2022-04-12T17:35:23.47+0530 lvl=info msg="Request Completed" method=POST path=/api/ds/query status=400 remote_addr=10.100.95.27 time_ms=36 size=65 referer=https://grafana.xxxx.xxxx.com/datasources/edit/6TjZwT87k
Solution

  • You cannot authenticate to the OpenShift prometheus instance using basic authentication. You need to authenticate using a bearer token, e.g. one obtained from oc whoami -t:

    curl -H "Authorization: Bearer $(oc whoami -t)" -k https://prometheus-k8s-openshift-monitoring.apps.xxxx.xxxx.xxxx.com/

    Or from a ServiceAccount with appropriate privileges:

    secret=$(oc -n openshift-monitoring get sa prometheus-k8s -o jsonpath='{.secrets[1].name}')
token=$(oc -n openshift-monitoring get secret $secret -o jsonpath='{.data.token}' | base64 -d)
curl -H "Authorization: Bearer $token" -k https://prometheus-k8s-openshift-monitoring.apps.xxxx.xxxx.xxxx.com/