How to solve Promtail extraScrapeConfigs not being picked up?

Question

It seems that excluding logs in a pod using the configuration below does not work.

extrascrapeconfig.yaml:

 - job_name: kubernetes-pods-app
   pipeline_stages:
   - docker: {}
   kubernetes_sd_configs:
   - role: pod
   relabel_configs:
   - action: drop
     regex: .+
     source_labels:
      - __meta_kubernetes_pod_label_name
    ###
   - action: keep
     regex: ambassador
     source_labels:
      - __meta_kubernetes_namespace
      - __meta_kubernetes_pod_namespace
      ###

To Reproduce

Steps to reproduce the behavior:

Deployed helm loki-stack :

helm install loki grafana/loki-stack --version "${HELM_CHART_VERSION}" \
  --namespace=monitoring \
  --create-namespace \
  -f "loki-stack-values-v${HELM_CHART_VERSION}.yaml"

loki-stack-values-v2.4.1.yaml:

loki:
  enabled: true
  config:


promtail:
  enabled: true
  extraScrapeConfigs: extrascrapeconfig.yaml

fluent-bit:
  enabled: false

grafana:
  enabled: false

prometheus:
  enabled: false

Attach grafana to loki datasource

Query: {namespace="kube-system"} in Grafana Loki

RESULT:

See logs

Expected behavior:

Not seeing any logs

Environment:

Infrastructure: Kubernetes
Deployment tool: Helm

What am I missing?

Answer 1

If you need Helm to pick up a specific file and pass it as a value, you should not pass the value itself in the values YAML file, but via another flag when installing or upgrading the release.

The command you are using is just applying the Helm values as-is, since the -f flag does not support parsing other files into the values by itself. Instead, use --set-file, which works similarly to --set, but gets the value content from the passed file.

Your command would now look like this:

    helm install loki grafana/loki-stack --version "${HELM_CHART_VERSION}" \
  --namespace=monitoring \
  --create-namespace \
  -f "loki-stack-values-v${HELM_CHART_VERSION}.yaml" \
  --set-file promtail.extraScrapeConfigs=extrascrapeconfig.yaml