Append results with fetch module

Question

I'm trying to pull in a file with the same name on multiple servers and I would like to just concatenate the results, but I don't think fetch module will allow me to do this. Can someone advise on another module that I could use for this task?

Current non-working code:

- hosts: '{{ target }}'
  gather_facts: false
  tasks:
    - name: Pull in file.log contents from servers, concatenating results
      fetch:
        src: '/tmp/file.log'
        dest: /tmp/fetched
        flat: yes
        fail_on_missing: no

Answer 1

For example, given the files

shell> ssh admin@test_11 cat /tmp/file.log
test_11
shell> ssh admin@test_12 cat /tmp/file.log
test_12
shell> ssh admin@test_13 cat /tmp/file.log
test_13

Throttle the task and time-stamp the fetched files, e.g.

- hosts: test_11,test_12,test_13
  tasks:
    - fetch:
        src: /tmp/file.log
        dest: /tmp/fetched/file-{{ time_stamp }}.log
        flat: true
        fail_on_missing: false
      throttle: 1
      vars:
        time_stamp: "{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"

gives

shell> tree /tmp/fetched/
/tmp/fetched/
├── file-2021-03-22_21-16-54.log
├── file-2021-03-22_21-16-58.log
└── file-2021-03-22_21-17-02.log

Then assemble the content of the files, e.g.

    - assemble:
        src: /tmp/fetched
        regexp: '^file-.*log$'
        dest: /tmp/fetched/assemble-{{ time_stamp }}.log
      vars:
        time_stamp: "{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"
      delegate_to: localhost
      run_once: true

gives

shell> cat /tmp/fetched/assemble-2021-03-22_21-17-07.log 
test_11
test_12
test_13

---

If you want to speed up the transfer from many hosts (e.g. ~100) increase the number of the parallel tasks (e.g. throttle: 10). Put the name of the host into the name of the file. Otherwise, the task would overwrite the files with the same timestamp, e.g.

    - fetch:
        src: /tmp/file.log
        dest: /tmp/fetched/file-{{ inventory_hostname }}-{{ time_stamp }}.log
        flat: true
        fail_on_missing: false
      throttle: 3
      vars:
        time_stamp: "{{ lookup('pipe', 'date +%Y-%m-%d_%H-%M-%S') }}"