Jinja2 & Ansible - Loop over lines in several text files

I'm using Ansible to retrieve a list of website we host over 50+ servers. I'm using a SQL query to store in a text file named as hostname_websites.txt using Ansible.

The next step is to build a prometheus (blackbox exporter) configuration file using that data. Each line of each text file represent an URL. For the sake of representation I'll declare that the URL is $URL and the hostname is $HOSTNAME.

I need some jinja2 magic to generate a single yaml file as :

{% for site in *** each text file *** %}
      - targets: ['{{ $URL }}']
        labels: {'node': '{{ $HOSTNAME }}'}
{% endfor %}

Given that in the end I need to have a config line containing each time the URL which is a line in text file, and the hostname as a label which is in the txt filename.

Every hostname is in a group called production in Ansible, so I tried looping over that group using jinja2 as follow:

{% for host in group['production'] %}
{{ lookup('file', '{{ host }}_websites.txt).splitlines()' }}
{% endfor %}

That gave me some jinja2 parsing error, as if it wasn't rendering the for loop at all. I stopped here, in the spirit that I would build by configuration skel around this statement.

Solution

There are two problems with your template that jump out immediately:

There is no variable group in Ansible (unless you've created one explicitly); you probably want groups.
You never nest Jinja {{...}} markers. If you want to generate a string containing a variable value, you can use string concatenation:
```
lookup('file', host ~ '_websites.txt')
```
Or string formatting:
```
lookup('file', '%s_websites.txt' % host)
```

Assuming that our inventory has a host node0 in the group production and that there exists a file node0_websites.txt with the following content:

http://stackoverflow.com/
http://ansible.com/
http://stackexchange.com/

Then running this playbook:

- hosts: localhost
  gather_facts: false
  tasks:
    - copy:
        dest: output.txt
        content: |
          {% for host in groups['production'] %}
          # Host: {{ host }}
          {% for url in lookup('file', host ~ '_websites.txt').splitlines() %}
          - {{ url }}
          {% endfor %}
          {% endfor %}

Generates the following content in output.txt:

# Host: node0
- http://stackoverflow.com/
- http://ansible.com/
- http://stackexchange.com/