Search code examples
fileansibleyamljinja2inventory

How to generate YAML/JSON from text file so I can use them as inventories for my playbook

My text (input.txt) file look like this, long list of server names with other info

it-linux1,p,a,3,3
it-test2,p,a,1,3
it-test3-dev,t,e,3,2
it-linux4,p,a,1,1
it-windows,t,e,3,4`

what I want to do is generate and inventories, for example an inventory for servers which have the p value or t and 3 values and so on I'm using the following playbook:

- name: Simple Read file and output JSON
  hosts: localhost
  connection: local
  become: false
  tasks:
  - name: Read Data File
    set_fact:
      data: "{{ lookup('ansible.builtin.file', '/home/checkout/myProjects/input.txt').split('\n') }}"

  - name: Show debug data
    ansible.builtin.debug: var=data

  - name: Create a new array
    set_fact:
      data_elements: []

  - name: Loop through lines and add data elements to array
    set_fact:
      data_elements: "{{ data_elements + [item] }}"
    loop: "{{ data }}"

  - name: Show debug data
    ansible.builtin.debug: var=data_elements

  - name: Write output YAML file
    copy:
      dest: output.yaml
      content: "{{ {'records': data_elements} | to_nice_yaml }}" ``

The output from this playbook:

TASK [Show debug data] ************************************************************************************************************************************
ok: [localhost] => {
"data": [


"it-linux1,p,a,3,3",
"it-test2,p,a,3,3",
"it-test3-dev,t,e,1,2",
"it-linux4,p,a,3,1 ",
"it-windows,t,e,3,4"
]
}

TASK [Create a new array] *********************************************************************************************************************************
ok: [localhost]

TASK [Loop through lines and add data elements to array] **************************************************************************************************
ok: [localhost] => (item=it-linux1,p,a,3,3)
ok: [localhost] => (item=it-test2,p,a,3,3)
ok: [localhost] => (item=it-test3-dev,t,e,1,2)
ok: [localhost] => (item=it-linux4,p,a,3,1 )
ok: [localhost] => (item=it-windows,t,e,3,4)

TASK [Show debug data] ************************************************************************************************************************************
ok: [localhost] => {
"data_elements": [
"it-linux1,p,a,3,3",
"it-test2,p,a,3,3",
"it-test3-dev,t,e,1,2",
"it-linux4,p,a,3,1 ",
"it-windows,t,e,3,4"
]
}

TASK [Write output json file] *****************************************************************************************************************************
changed: [localhost]

The playbook generate a YAML file (output.yaml) like this :

records:
- it-linux1,p,a,3,3
- it-test2,p,a,3,3
- it-test3-dev,t,e,1,2
- it-linux4,p,a,3,1
- it-windows,t,e,3,4

Is there a way to read and process these values (output.yaml) to generate an inventories like an inventory for all servers that have p and 3 and another for servers that have p and 1 values?

Solution

  • Given the file

    shell> cat input.txt 
it-linux1,p,a,3,3
it-test2,p,a,1,3
it-test3-dev,t,e,3,2
it-linux4,p,a,1,1
it-windows,t,e,3,4

    Q: Generate inventories for servers that have p and 3 and another for servers that have p and 1 values."

    A: Use the module community.general.read_csv to read the file

        - community.general.read_csv:
        path: "{{ playbook_dir }}/input.txt"
        fieldnames: host,v1,v2,v3,v4
      register: data
    - debug:
        var: data.list|to_yaml

    gives

      data.list:
    - {host: it-linux1, v1: p, v2: a, v3: '3', v4: '3'}
    - {host: it-test2, v1: p, v2: a, v3: '1', v4: '3'}
    - {host: it-test3-dev, v1: t, v2: e, v3: '3', v4: '2'}
    - {host: it-linux4, v1: p, v2: a, v3: '1', v4: '1'}
    - {host: it-windows, v1: t, v2: e, v3: '3', v4: '4'}

    There are three options on how to create the inventory:

    • Create inventory files host-p1, host-p3, ... and use them later
    • Use the module add_host and create the in-memory inventory groups
    • Use inventory plugin ansible.builtin.constructed to create the inventory groups
    1. Create the inventory files for values p,3 and p,1. Declare the dictionary
      my_groups:
    p3: '[?(v1 == `p`) && (to_number(v3) == `3` || to_number(v4) == `3`)]'
    p1: '[?(v1 == `p`) && (to_number(v3) == `1` || to_number(v4) == `1`)]'

    and create the files

        - name: Inventory files
      copy:
        dest: "{{ playbook_dir }}/hosts-{{ item.key }}"
        content: |
          [{{ item.key }}]
          {% for i in data.list|json_query(item.value) %}
          {{ i.host }}
          {% endfor %}
      loop: "{{ my_groups|dict2items }}"

    shell> cat hosts-p3 
[p3]
it-linux1
it-test2

    shell> cat hosts-p1
[p1]
it-test2
it-linux4

    Example of a complete playbook for testing

    - hosts: localhost

  vars:

    my_groups:
      p3: '[?(v1 == `p`) && (to_number(v3) == `3` || to_number(v4) == `3`)]'
      p1: '[?(v1 == `p`) && (to_number(v3) == `1` || to_number(v4) == `1`)]'

  tasks:

    - community.general.read_csv:
        path: "{{ playbook_dir }}/input.txt"
        fieldnames: host,v1,v2,v3,v4
      register: data
    - debug:
        var: data.list|to_yaml

    - name: Inventory files
      copy:
        dest: "{{ playbook_dir }}/hosts-{{ item.key }}"
        content: |
          [{{ item.key }}]
          {% for i in data.list|json_query(item.value) %}
          {{ i.host }}
          {% endfor %}
      loop: "{{ my_groups|dict2items }}"
    1. Use the module add_host and create the in-memory inventory groups. Create the list of groups and hosts. Declare the variables
      my_groups_hosts_str: |
    [{% for k,v in my_groups.items() %}
    {group: {{ k }}, hosts: {{ data.list|json_query(v)|map(attribute='host') }}},
    {% endfor %}]
  my_groups_hosts: "{{ my_groups_hosts_str|from_yaml }}"

    gives

      my_groups_hosts:
    - group: p3
      hosts: [it-linux1, it-test2]
    - group: p1
      hosts: [it-test2, it-linux4]

    Create the in-memory inventory groups

        - name: In-memory inventory groups
      add_host:
        groups: "{{ item.0.group }}"
        hostname: "{{ item.1 }}"
      with_subelements:
        - "{{ my_groups_hosts|from_yaml }}"
        - hosts

    Display all groups

    - hosts: all
  tasks:
    - debug:
        msg: |
          {% for g in groups %}
          {{ g }}: {{ groups[g] }}
          {% endfor %}
      run_once: true

    gives (abridged)

    TASK [debug] ************************************************************
ok: [it-linux1] =>
  msg:
    all: ['it-linux1', 'it-test2', 'it-linux4', 'localhost']
    ungrouped: ['localhost']
    p3: ['it-linux1', 'it-test2']
    p1: ['it-test2', 'it-linux4']

    Use the group p3 in a playbook

    - hosts: p3
  tasks:
    - debug:
        var: inventory_hostname

    gives (abridged)

    TASK [debug] ************************************************************
ok: [it-linux1] => 
  inventory_hostname: it-linux1
ok: [it-test2] => 
  inventory_hostname: it-test2

    Example of a complete playbook for testing

    - name: In-memory inventory groups
  hosts: localhost

  vars:

    my_groups:
      p3: '[?(v1 == `p`) && (to_number(v3) == `3` || to_number(v4) == `3`)]'
      p1: '[?(v1 == `p`) && (to_number(v3) == `1` || to_number(v4) == `1`)]'

    my_groups_hosts_str: |
      [{% for k,v in my_groups.items() %}
      {group: {{ k }}, hosts: {{ data.list|json_query(v)|map(attribute='host') }}},
      {% endfor %}]
    my_groups_hosts: "{{ my_groups_hosts_str|from_yaml }}"

  tasks:

    - community.general.read_csv:
        path: "{{ playbook_dir }}/input.txt"
        fieldnames: host,v1,v2,v3,v4
      register: data
    - debug:
        var: data.list|to_yaml
    - debug:
        var: my_groups_hosts|to_yaml

    - name: In-memory inventory groups
      add_host:
        groups: "{{ item.0.group }}"
        hostname: "{{ item.1 }}"
      with_subelements:
        - "{{ my_groups_hosts|from_yaml }}"
        - hosts

- hosts: all
  tasks:
    - debug:
        msg: |
          {% for g in groups %}
          {{ g }}: {{ groups[g] }}
          {% endfor %}
      run_once: true

- hosts: p3
  tasks:
    - debug:
        var: inventory_hostname
    1. Use the inventory plugin ansible.builtin.constructed to create the inventory groups

    See details about the plugin

    shell> ansible-doc -t inventory ansible.builtin.constructed

    Create the directory inventory and create the file 01-hosts with all hosts and all variables. For example,

    shell> cat pb-create-hosts.yml
- name: Create inventory/01-hosts
  hosts: localhost

  vars:

    fieldnames: host,v1,v2,v3,v4
    filednames_list: "{{ fieldnames.split(',') }}"

  tasks:

    - community.general.read_csv:
        path: "{{ playbook_dir }}/input.txt"
        fieldnames: "{{ fieldnames }}"
      register: data
    - debug:
        var: data.list|to_yaml

    - name: Create inventory/01-hosts
      copy:
        dest: "{{ playbook_dir }}/inventory/01-hosts"
        content: |
          {% for i in data.list %}
          {{ i.host }} {% for j in filednames_list[1:] %}
          {{ j }}={{ i[j] }} {% endfor %}

          {% endfor %}

    will create the inventory file in the INI format

    shell> cat inventory/01-hosts 
it-linux1 v1=p v2=a v3=3 v4=3 
it-test2 v1=p v2=a v3=1 v4=3 
it-test3-dev v1=t v2=e v3=3 v4=2 
it-linux4 v1=p v2=a v3=1 v4=1 
it-windows v1=t v2=e v3=3 v4=4

    Create the configuration file for the plugin ansible.builtin.constructed

    shell> cat inventory/02-constructed.yml
plugin: ansible.builtin.constructed
strict: true
use_vars_plugins: true
use_extra_vars: true
groups:
  p1: "v1 == 'p' and (v3 == 1 or v4 == 1)"
  p3: "v1 == 'p' and (v3 == 3 or v4 == 3)"

    and test the inventory

    shell> ansible-inventory -i inventory --list --yaml
all:
  children:
    p1:
      hosts:
        it-linux4:
          v1: p
          v2: a
          v3: 1
          v4: 1
        it-test2: {}
    p3:
      hosts:
        it-linux1:
          v1: p
          v2: a
          v3: 3
          v4: 3
        it-test2:
          v1: p
          v2: a
          v3: 1
          v4: 3
    ungrouped:
      hosts:
        it-test3-dev:
          v1: t
          v2: e
          v3: 3
          v4: 2
        it-windows:
          v1: t
          v2: e
          v3: 3
          v4: 4

    Use the groups in a playbook. For example,

    shell> cat pb-p3.yml
- hosts: p3
  tasks:
    - debug:
        var: inventory_hostname

    gives (abridged)

    shell> ansible-playbook -i inventory pb-p3.yml
  ...
TASK [debug] ************************************************************
ok: [it-linux1] => 
  inventory_hostname: it-linux1
ok: [it-test2] => 
  inventory_hostname: it-test2