Search code examples
promtail

How to set default value for regular in Promtail configueration


I have Docker swarm of 3 nodes where are running monitoring apps (exporters, grafana, etc..) per evironment (dev, stg and prd). Dev an stg containers contains _<env>_ in their container name, so for dev its _dev_ and for stg its _stg_. Prd containers does not have environment contained in the name of container.

I am parsing with regular expression from the name the environment and setting it up as label:

  relabel_configs:
    ...
    - source_labels: ['__meta_docker_container_name']
      regex: '/.*_(dev|stg)_.*'
      target_label: 'env'

I would like to set prd as default environment. I tried to set it as static_label:

  pipeline_stages:
    - static_labels:
        env: prd

But then, all containers has env label set to prd. It looks like relabel config is used first and then static labels are applied. Is there a way how to configure it properly?


Solution

  • So, its looks like relabel_configs is executed before pipeline_stages. So firstly, I parsed dev or stg from container name with this configuration:

    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/.*_(dev|stg)_.*'
        target_label: 'env'
    

    And than, using template stage with go template if condition check, if label env exists, keep it. If not, create it with default value prd:

    pipeline_stages:
      - template:
          source: env
          template: '{{ if .Value }}{{ .Value }}{{ else }}prd{{ end }}'
      - labels:
          env:
    

    Also important section is labels. Stage template will only create key env with value prd (if does not exists from previous step), but does not create label from it. Thats why there have to be this labels stage (label stage create labels from keys in the message).

    This is my final pipeline:

    scrape_configs:
    - job_name: docker
      docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 30s
      relabel_configs:
        - source_labels: ['__meta_docker_container_name']
          regex: '/.*\.([0-9]{1,2})\..*'
          target_label: 'replica'
        - source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
          regex: '(.*)'
          target_label: 'service'
        - source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
          regex: '(.*)'
          target_label: 'stack'
        - source_labels: ['__meta_docker_container_name']
          regex: '/.*_(dev|stg)_.*'
          target_label: 'env'
      pipeline_stages:
        - template:
            source: env
            template: '{{ if .Value }}{{ .Value }}{{ else }}prd{{ end }}'
        - labels:
            env:
        - static_labels:
            hostname: ${NODE_HOSTNAME}
    

    Variable NODE_HOSTNAME is passed to the container from node where container is running on. So Its hostname one of node name from swarm.