Search code examples
ansiblesplunk

Ansible - Creating Splunk Indexes while skipping already existing ones

I am currently attempting to create a script to generate index names based on 2 variables: a prefix based on a name ingested when I run the playbook (entity_name/ group_name etc), and a list of suffixes defined in the playbook ( data sources that I would like to monitor; IPS, WAF, Firewall etc).

I would like to write into the index.conf file and fill out index names and file paths. The problem I am facing is checking if the index already exists (entity_name_waf exists, for example) and skip the index, while adding in lines for the indexes that currently do not exist (entity_name_ips would be created as it does not exist, for example).

My main problem would be comparing the existing entries to the ones I am attempting to add. I have tried to use the match() function to check if the variable exists, but it does not allow me and results in a "variable is not defined" error.

what I expect:

index.conf:

[AAA_1]
homePath = $SPLUNK_DB/AAA_1/db
coldPath = $SPLUNK_DB/AAA_1/colddb
thawedPath = $SPLUNK_DB/AAA_1/thaweddb
repFactor = auto
[AAA_2]
homePath = $SPLUNK_DB/AAA_2/db
coldPath = $SPLUNK_DB/AAA_2/colddb
thawedPath = $SPLUNK_DB/AAA_2/thaweddb
repFactor = auto
[AAA_3]
homePath = $SPLUNK_DB/AAA_3/db
coldPath = $SPLUNK_DB/AAA_3/colddb
thawedPath = $SPLUNK_DB/AAA_3/thaweddb
repFactor = auto

running the playbook with AAA as input:

'debug: AAA_1 already exists! skipping.' 'debug: AAA_2 already exists! skipping.' 'debug: AAA_3 already exists! skipping.' 'debug: creating AAA_4...' 'debug: creating AAA_5...' . . .

resulting index.conf:

[AAA_1]
homePath = $SPLUNK_DB/AAA_1/db
coldPath = $SPLUNK_DB/AAA_1/colddb
thawedPath = $SPLUNK_DB/AAA_1/thaweddb
repFactor = auto
[AAA_2]
homePath = $SPLUNK_DB/AAA_2/db
coldPath = $SPLUNK_DB/AAA_2/colddb
thawedPath = $SPLUNK_DB/AAA_2/thaweddb
repFactor = auto
[AAA_3]
homePath = $SPLUNK_DB/AAA_3/db
coldPath = $SPLUNK_DB/AAA_3/colddb
thawedPath = $SPLUNK_DB/AAA_3/thawddb
repFactor = auto
[AAA_4]
homePath = $SPLUNK_DB/AAA_4/db
coldPath = $SPLUNK_DB/AAA_4/colddb
thawedPath = $SPLUNK_DB/AAA_4/thaweddb
repFactor = auto
[AAA_5]
homePath = $SPLUNK_DB/AAA_5/db
coldPath = $SPLUNK_DB/AAA_5/colddb
thawedPath = $SPLUNK_DB/AAA_5/thaweddb
repFactor = auto

edit:

regarding prefix variable, I define it (is that the term?) when I run the playbook. for example, ansible-playbook test_playbook -e '{"prefix":"entity_name"}'

as for the suffix, I defined it as a list before any tasks are run.

suffix: 
   - suffix_1 
   - suffix_2 
   - suffix_3

I then read the index.conf file and register it shell: cat /home/splunk/index.conf register: index_file

Here is where I think im getting my first problem. I need to combine the prefix and suffix into a single variable and use it in the rest of the playbook, but if I define it in one task, it becomes undefined again in the next task.

- name: generate combined list
  index_name: "{{ prefix }}_{{ item }}"
  loop: "{{ suffix }}"
Solution

  • First create a list with prefix and suffix combined. A good example from Doug How to concatenate with a string each element of a list in ansible:

    - name: generate combined list
  set_fact: 
    combined_prefix_suffix: "{{ [prefix] | product(suffix) | map('join') | list }}"

    This gives:

    Combined_prefix_suffix: [
    "AAA_1",
    "AAA_2",
    "AAA_3"
 ]

    Use blockinfile module to insert block texts together with marker to append new block texts and avoid overriding.

    - name: Add block text 
  blockinfile:
    path: "{{ index_path }}"
    block: |
      [{{ item }}]
      homePath = $SPLUNK_DB/{{ item }}/db
      coldPath = $SPLUNK_DB/{{ item }}/colddb
      thawedPath = $SPLUNK_DB/{{ item }}/thaweddb
      repFactor = auto

  marker: "## {mark} added by ansible {{ item }}"
  register: output
  loop: "{{ combined_prefix_suffix }}"

    With below vars gives:

    vars:
  index_path: WAF/index.conf
  suffix:
    - _1
    - _2
    - _3
    # - _4
    # - _5


## BEGIN added by ansible AAA_1
[AAA_1]
homePath = $SPLUNK_DB/AAA_1/db
coldPath = $SPLUNK_DB/AAA_1/colddb
thawedPath = $SPLUNK_DB/AAA_1/thaweddb
repFactor = auto
## END added by ansible AAA_1
## BEGIN added by ansible AAA_2
[AAA_2]
homePath = $SPLUNK_DB/AAA_2/db
coldPath = $SPLUNK_DB/AAA_2/colddb
thawedPath = $SPLUNK_DB/AAA_2/thaweddb
repFactor = auto
## END added by ansible AAA_2
## BEGIN added by ansible AAA_3
[AAA_3]
homePath = $SPLUNK_DB/AAA_3/db
coldPath = $SPLUNK_DB/AAA_3/colddb
thawedPath = $SPLUNK_DB/AAA_3/thaweddb
repFactor = auto
## END added by ansible AAA_3

    With below var gives:

    vars:
  index_path: WAF/index.conf
  suffix:
    - _1
    - _2
    - _3
    - _4
    - _5


## BEGIN added by ansible AAA_1
[AAA_1]
homePath = $SPLUNK_DB/AAA_1/db
coldPath = $SPLUNK_DB/AAA_1/colddb
thawedPath = $SPLUNK_DB/AAA_1/thaweddb
repFactor = auto
## END added by ansible AAA_1
## BEGIN added by ansible AAA_2
[AAA_2]
homePath = $SPLUNK_DB/AAA_2/db
coldPath = $SPLUNK_DB/AAA_2/colddb
thawedPath = $SPLUNK_DB/AAA_2/thaweddb
repFactor = auto
## END added by ansible AAA_2
## BEGIN added by ansible AAA_3
[AAA_3]
homePath = $SPLUNK_DB/AAA_3/db
coldPath = $SPLUNK_DB/AAA_3/colddb
thawedPath = $SPLUNK_DB/AAA_3/thaweddb
repFactor = auto
## END added by ansible AAA_3
## BEGIN added by ansible AAA_4
[AAA_4]
homePath = $SPLUNK_DB/AAA_4/db
coldPath = $SPLUNK_DB/AAA_4/colddb
thawedPath = $SPLUNK_DB/AAA_4/thaweddb
repFactor = auto
## END added by ansible AAA_4
## BEGIN added by ansible AAA_5
[AAA_5]
homePath = $SPLUNK_DB/AAA_5/db
coldPath = $SPLUNK_DB/AAA_5/colddb
thawedPath = $SPLUNK_DB/AAA_5/thaweddb
repFactor = auto
## END added by ansible AAA_5

    Example of playbook for testing:

    - hosts: localhost
  vars:
    index_path: WAF/index.conf
    suffix:
      - _1
      - _2
      - _3
      # - _4
      # - _5
  tasks:
    - name: generate combined list
      set_fact: 
        combined_prefix_suffix: "{{ [prefix] | product(suffix) | map('join') | list }}"
    
    - debug: 
        msg: "{{ combined_prefix_suffix }}"
    
    - name: Add block text 
      blockinfile:
        path: "{{ index_path }}"
        block: |
          [{{ item }}]
          homePath = $SPLUNK_DB/{{ item }}/db
          coldPath = $SPLUNK_DB/{{ item }}/colddb
          thawedPath = $SPLUNK_DB/{{ item }}/thaweddb
          repFactor = auto
        marker: "## {mark} added by ansible {{ item }}"
        
      register: output
      loop: "{{ combined_prefix_suffix }}"

    Update:

    If there are already indexes that exist and might have been added manually and those are not managed or were not added by Ansible, you can use lineinfile to check the lines. Below I demonstrate how to achieve this.

    Here is the complete playbook for testing purposes.

    - hosts: localhost
  vars:
    index_path: WAF/index.conf
    suffix:
      - _1
      - _2
      - _3
      - _4
      - _5
  tasks:
    - name: generate combined list
      set_fact: 
        combined_prefix_suffix: "{{ [prefix] | product(suffix) | map('join') | list }}"
    
    - name: output combined list
      debug: 
        msg: "{{ combined_prefix_suffix }}"

    - name: check if index is present in file
      lineinfile:
        path: "{{ index_path }}"
        regexp: '\[{{ item }}\]'
        state: absent
      check_mode: True
      changed_when: False
      register: check_line
      loop: "{{ combined_prefix_suffix }}"

    - name: output if line exists
      debug:
        msg: "{{ item.item }} already exists! skipping"
      loop: "{{ check_line.results }}"
      loop_control:
        label: "{{ item.found }}"
      when: item.found
    
    - name: output if line does not exist
      debug:
        msg: "{{ item.item }} does not exist. {{ item.item }} will be added in the next task"
      loop: "{{ check_line.results }}"
      loop_control:
        label: "{{ item.found }}"
      when: not item.found

    
    - name: Add block text only if index does not exist in file
      blockinfile:
        path: "{{ index_path }}"
        block: |           
          [{{ item.0 }}]
          homePath = $SPLUNK_DB/{{ item.0 }}/db
          coldPath = $SPLUNK_DB/{{ item.0 }}/colddb
          thawedPath = $SPLUNK_DB/{{ item.0 }}/thaweddb
          repFactor = auto
        marker: "## {mark} added by ansible {{ item.0 }}"

      register: output
      with_together: 
        - "{{ combined_prefix_suffix }}"
        - "{{ check_line.results }}"
      loop_control:
        label: "{{ item.1.found }}"
      when: not item.1.found

    Where index AAA_1, AAA_2 and AAA_3 were already in the file. This gives and appends the missing two indexes without touching the existing indexes.

    [AAA_1]
homePath = $SPLUNK_DB/AAA_1/db
coldPath = $SPLUNK_DB/AAA_1/colddb
thawedPath = $SPLUNK_DB/AAA_1/thaweddb
repFactor = auto
[AAA_2]
homePath = $SPLUNK_DB/AAA_2/db
coldPath = $SPLUNK_DB/AAA_2/colddb
thawedPath = $SPLUNK_DB/AAA_2/thaweddb
repFactor = auto
[AAA_3]
homePath = $SPLUNK_DB/AAA_3/db
coldPath = $SPLUNK_DB/AAA_3/colddb
thawedPath = $SPLUNK_DB/AAA_3/thaweddb
repFactor = auto
## BEGIN added by ansible AAA_4
[AAA_4]
homePath = $SPLUNK_DB/AAA_4/db
coldPath = $SPLUNK_DB/AAA_4/colddb
thawedPath = $SPLUNK_DB/AAA_4/thaweddb
repFactor = auto
## END added by ansible AAA_4
## BEGIN added by ansible AAA_5
[AAA_5]
homePath = $SPLUNK_DB/AAA_5/db
coldPath = $SPLUNK_DB/AAA_5/colddb
thawedPath = $SPLUNK_DB/AAA_5/thaweddb
repFactor = auto
## END added by ansible AAA_5

    cli outputs:

    PLAY [localhost] ***************************************************************************************************************************************************************************

TASK [Gathering Facts] *********************************************************************************************************************************************************************
ok: [localhost]

TASK [generate combined list] **************************************************************************************************************************************************************
ok: [localhost]

TASK [output combined list] ****************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": [
        "AAA_1",
        "AAA_2",
        "AAA_3",
        "AAA_4",
        "AAA_5"
    ]
}

TASK [check if index is present in file] ***************************************************************************************************************************************************
ok: [localhost] => (item=AAA_1)
ok: [localhost] => (item=AAA_2)
ok: [localhost] => (item=AAA_3)
ok: [localhost] => (item=AAA_4)
ok: [localhost] => (item=AAA_5)

TASK [output if line exists] ***************************************************************************************************************************************************************
ok: [localhost] => (item=1) => {
    "msg": "AAA_1 already exists! skipping"
}
ok: [localhost] => (item=1) => {
    "msg": "AAA_2 already exists! skipping"
}
ok: [localhost] => (item=1) => {
    "msg": "AAA_3 already exists! skipping"
}
skipping: [localhost] => (item=0) 
skipping: [localhost] => (item=0) 

TASK [output if line does not exist] *******************************************************************************************************************************************************
skipping: [localhost] => (item=1) 
skipping: [localhost] => (item=1) 
skipping: [localhost] => (item=1) 
ok: [localhost] => (item=0) => {
    "msg": "AAA_4 does not exist. AAA_4 will be added in the next task"
}
ok: [localhost] => (item=0) => {
    "msg": "AAA_5 does not exist. AAA_5 will be added in the next task"
}

TASK [Add block text only if index does not exist in file] *********************************************************************************************************************************
skipping: [localhost] => (item=1) 
skipping: [localhost] => (item=1) 
skipping: [localhost] => (item=1) 
changed: [localhost] => (item=0)
changed: [localhost] => (item=0)

PLAY RECAP *********************************************************************************************************************************************************************************
localhost                  : ok=7    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

    Update2 According to the last comments about having special index or indexes this should do the job. Note that multiple changes were done and I created another variable called special_suffix to avoid hardcoding for only one index.

    Example of playbook:

    - hosts: localhost
  vars:
    index_path: WAF/index.conf
    suffix:
      - _1
      - _2
      - _3
      - _4
      - _5
    special_suffix:
      - _6
  tasks:
    - name: generate combined list
      set_fact: 
        combined_prefix_suffix: "{{ [prefix] | product(suffix) | map('join') | list }}"
      when:
        - prefix is defined
        - suffix is defined

    - name: generate special combined list
      set_fact: 
        combined_special_prefix_suffix: "{{ [special_prefix] | product(special_suffix) | map('join') | list }}"
      when: 
        - special_prefix is defined
        - special_suffix is defined

    - name: output combined_prefix_suffix list
      debug: 
        msg: 
          - "{{ combined_prefix_suffix }}"
      when:
        - combined_prefix_suffix is defined

    - name: output combined_special_prefix_suffix combined list
      debug: 
        msg: 
          - "{{ combined_special_prefix_suffix }}"
      when:
        - combined_special_prefix_suffix is defined

    - name: check if index is present in file
      lineinfile:
        path: "{{ index_path }}"
        regexp: '\[{{ item }}\]'
        state: absent
      check_mode: True
      changed_when: False
      register: check_line
      loop: "{{ combined_prefix_suffix }}"
      when:
        - combined_prefix_suffix is defined

    - name: check if special index is present in file
      lineinfile:
        path: "{{ index_path }}"
        regexp: '\[{{ item }}\]'
        state: absent
      check_mode: True
      changed_when: False
      register: check_special_index_line
      loop: "{{ combined_special_prefix_suffix }}"
      when:
        - combined_special_prefix_suffix is defined

    - name: output if line exists
      debug:
        msg: "{{ item.item }} index already exists! skipping"
      loop: "{{ check_line.results }}"
      loop_control:
        label: "{{ item.found }}"
      when: 
        - combined_prefix_suffix is defined
        - item.found
    
    - name: output if special line exists
      debug:
        msg: "{{ item.item }} special index already exists! skipping"
      loop: "{{ check_special_index_line.results }}"
      loop_control:
        label: "{{ item.found }}"
      when: 
        - combined_special_prefix_suffix is defined
        - item.found

    - name: output if line does not exist
      debug:
        msg: "{{ item.item }} does not exist. {{ item.item }} will be added in the next task"
      loop: "{{ check_line.results }}"
      loop_control:
        label: "{{ item.found }}"
      when: 
        - combined_prefix_suffix is defined
        - not item.found

    - name: output if special line does not exist
      debug:
        msg: "{{ item.item }} does not exist. {{ item.item }} will be added in the next task"
      loop: "{{ check_special_index_line.results }}"
      loop_control:
        label: "{{ item.found }}"
      when:
        - combined_special_prefix_suffix is defined
        - not item.found
    
    - name: Add block text only if index does not exist in file
      blockinfile:
        path: "{{ index_path }}"
        block: |           
          [{{ item.0 }}]
          homePath = $SPLUNK_DB/{{ item.0 }}/db
          coldPath = $SPLUNK_DB/{{ item.0 }}/colddb
          thawedPath = $SPLUNK_DB/{{ item.0 }}/thaweddb
          repFactor = auto
        marker: "## {mark} added by ansible {{ item.0 }}"

      register: output
      with_together: 
        - "{{ combined_prefix_suffix }}"
        - "{{ check_line.results }}"
      loop_control:
        label: "{{ item.1.found }}"
      when:
        - combined_prefix_suffix is defined
        - not item.1.found

    - name: Add block text only if index is special and does not exist in file
      blockinfile:
        path: "{{ index_path }}"
        block: |           
          [{{ item.0 }}]
          homePath = $SPLUNK_DB/{{ item.0 }}/db
          coldPath = $SPLUNK_DB/{{ item.0 }}/colddb
          thawedPath = $SPLUNK_DB/{{ item.0 }}/thaweddb
          frozenTimePeriodInSecs = $SPLUNK_DB/{{ item.0 }}/example
          repFactor = auto
        marker: "## {mark} added by ansible {{ item.0 }}"

      register: output
      with_together: 
        - "{{ combined_special_prefix_suffix }}"
        - "{{ check_special_index_line.results }}"
      loop_control:
        label: "{{ item.1.found }}"
      when: 
        - combined_special_prefix_suffix is defined
        - not item.1.found

    This gives:

    [AAA_1]
homePath = $SPLUNK_DB/AAA_1/db
coldPath = $SPLUNK_DB/AAA_1/colddb
thawedPath = $SPLUNK_DB/AAA_1/thaweddb
repFactor = auto
[AAA_2]
homePath = $SPLUNK_DB/AAA_2/db
coldPath = $SPLUNK_DB/AAA_2/colddb
thawedPath = $SPLUNK_DB/AAA_2/thaweddb
repFactor = auto
[AAA_3]
homePath = $SPLUNK_DB/AAA_3/db
coldPath = $SPLUNK_DB/AAA_3/colddb
thawedPath = $SPLUNK_DB/AAA_9/thaweddb
repFactor = auto
## BEGIN added by ansible AAA_4
[AAA_4]
homePath = $SPLUNK_DB/AAA_4/db
coldPath = $SPLUNK_DB/AAA_4/colddb
thawedPath = $SPLUNK_DB/AAA_4/thaweddb
repFactor = auto
## END added by ansible AAA_4
## BEGIN added by ansible AAA_5
[AAA_5]
homePath = $SPLUNK_DB/AAA_5/db
coldPath = $SPLUNK_DB/AAA_5/colddb
thawedPath = $SPLUNK_DB/AAA_5/thaweddb
repFactor = auto
## END added by ansible AAA_5

## BEGIN added by ansible AAA_6
[AAA_6]
homePath = $SPLUNK_DB/AAA_6/db
coldPath = $SPLUNK_DB/AAA_6/colddb
thawedPath = $SPLUNK_DB/AAA_6/thaweddb
frozenTimePeriodInSecs = $SPLUNK_DB/AAA_6/example
repFactor = auto
## END added by ansible AAA_6