I have Docker swarm of 3 nodes where are running monitoring apps (exporters, grafana, etc..) per evironment (dev, stg and prd). Dev an stg containers contains _<env>_
in their container name, so for dev its _dev_
and for stg its _stg_
. Prd containers does not have environment contained in the name of container.
I am parsing with regular expression from the name the environment and setting it up as label:
relabel_configs:
...
- source_labels: ['__meta_docker_container_name']
regex: '/.*_(dev|stg)_.*'
target_label: 'env'
I would like to set prd as default environment. I tried to set it as static_label
:
pipeline_stages:
- static_labels:
env: prd
But then, all containers has env label set to prd. It looks like relabel config is used first and then static labels are applied. Is there a way how to configure it properly?
So, its looks like relabel_configs
is executed before pipeline_stages
. So firstly, I parsed dev or stg from container name with this configuration:
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/.*_(dev|stg)_.*'
target_label: 'env'
And than, using template stage with go template if condition check, if label env
exists, keep it. If not, create it with default value prd:
pipeline_stages:
- template:
source: env
template: '{{ if .Value }}{{ .Value }}{{ else }}prd{{ end }}'
- labels:
env:
Also important section is labels
. Stage template will only create key env with value prd (if does not exists from previous step), but does not create label from it. Thats why there have to be this labels stage (label stage create labels from keys in the message).
This is my final pipeline:
scrape_configs:
- job_name: docker
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 30s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/.*\.([0-9]{1,2})\..*'
target_label: 'replica'
- source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
regex: '(.*)'
target_label: 'service'
- source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
regex: '(.*)'
target_label: 'stack'
- source_labels: ['__meta_docker_container_name']
regex: '/.*_(dev|stg)_.*'
target_label: 'env'
pipeline_stages:
- template:
source: env
template: '{{ if .Value }}{{ .Value }}{{ else }}prd{{ end }}'
- labels:
env:
- static_labels:
hostname: ${NODE_HOSTNAME}
Variable NODE_HOSTNAME
is passed to the container from node where container is running on. So Its hostname one of node name from swarm.