How do I fetch multiple files from hosts with varying paths in Ansible?

Question

I am attempting to write a script that will fetch files from multiple hosts, the paths vary across each remote host as well as the number of files to fetch from each one.

My issue is pertaining to how I can loop through/iterate over each list of files when they are not of fixed length from within a with_items block?

Since the number of files and paths vary I have declared my inventory file as follows to make my playbook more flexible. Each fetch_path from the remote host corresponds to a land_path on the localhost:

[hosts]
host1 fetch_paths='["path/to/file1", "path/to/file2"]' land_paths='["path/to/file1", "path/to/file2"]'

host2 fetch_paths='["path/to/file1"]' land_paths='["path/to/file1"]'

host3 fetch_paths='["path/to/file1", "path/to/file2", "path/to/file3"]' land_paths='["path/to/file1", "path/to/file2", "path/to/file3"]' 

My current playbook is as follows but I am not sure how I can nest a loop statement within my with_items statement AND have it relate the fetch_path to a land_path. I assume I will need some sort of index iterator based on the length of each list but I am unsure of how and where this logic should be implemented.

---

- name: fetch files from remote host then execute batch scripts on the localhost
  hosts: hosts
  tasks:
    - name: fetch files from remote host to local
    fetch:
      src: "{{ item.fetch_paths }}"
      dest: "{{ item.land_paths }}"
      flat: yes
      fail_on_missing: yes
      validate_certs: no
    with_items: "{{ groups['hosts'] }}"

I am very new to Ansible (this is my first playbook) so any guidance in the right direction is much appreciated, thanks in advance!

Answer 1

zip the lists. For example, the play below

shell> cat pb.yml
- hosts: all

  tasks:

    - fetch:
        src: "{{ item.0 }}"
        dest: "{{ item.1 }}"
        flat: true
      loop: "{{ fetch_paths|zip(land_paths) }}"

given the inventory

shell> cat hosts
host1
host2

and the host_vars

shell> cat host_vars/host1
fetch_paths: [/etc/passwd, /etc/services]
land_paths: [/tmp/host1/etc/passwd, /tmp/host1/etc/services]

shell> cat host_vars/host2
fetch_paths: [/etc/passwd]
land_paths: [/tmp/host2/etc/passwd]

will fetch the files as expected

shell> ansible-playbook pb.yml

PLAY [all] ************************************************************************************

TASK [fetch] **********************************************************************************
changed: [host2] => (item=['/etc/passwd', '/tmp/host2/etc/passwd'])
changed: [host1] => (item=['/etc/passwd', '/tmp/host1/etc/passwd'])
changed: [host1] => (item=['/etc/services', '/tmp/host1/etc/services'])

PLAY RECAP ************************************************************************************
host1: ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host2: ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

shell> tree /tmp/host1
/tmp/host1
└── etc
    ├── passwd
    └── services

1 directory, 2 files

shell> tree /tmp/host2
/tmp/host2
└── etc
    └── passwd

1 directory, 1 file

---

Note 1

It's not documented that the parameter dest can be a file

dest: A directory to save the file into. For example, if the dest directory is /backup a src file named /etc/profile on host host.example.com, would be saved into /backup/host.example.com/etc/profile. The hostname is based on the inventory name.

See: The parameter flat

---

Note 2

You can simplify the workflow if you don't change the pathnames of the fetched files. The module fetch by default adds the hostname to the path. Quoting from dest:

A directory to save the file into. For example, if the dest' directory is /backup' a src' file named /etc/profile' on host host.example.com', would be saved into /backup/host.example.com/etc/profile'. The hostname is based on the inventory name.

The below play

- hosts: all

  tasks:

    - fetch:
        src: "{{ item }}"
        dest: /tmp/ansible_fetch
      loop: "{{ fetch_paths }}"

puts the fetched files into the directory /tmp/ansible_fetch (change it to /tmp if you want to) on the controller

shell> tree /tmp/ansible_fetch/
/tmp/ansible_fetch/
├── host1
│   └── etc
│       ├── passwd
│       └── services
└── host2
    └── etc
        └── passwd

As a consequence, you don't need the lists land_paths anymore

shell> cat host_vars/host1
fetch_paths: [/etc/passwd, /etc/services]

shell> cat host_vars/host2
fetch_paths: [/etc/passwd]