Switching user for delegation to host outside of inventory with Ansible/awx

I am trying to do the following using Ansible 2.8.4 and awx:

Read some facts from Cisco IOS devices (works)
Put results into a local file using a template (works)
Copy/Move the resulting file to a different server

Since I have to use a different user to access IOS devices and servers, and the servers in question aren't part of the inventory used for the playbook, I am trying to achieve this using become_user and delegate_to. The initial user (defined in the awx template) is allowed to connect to the IOS devices, while different_user can connect to servers using a ssh private key.

The playbook:

---
  - name: Read Switch Infos
    hosts: all
    gather_facts: no
    tasks:

      - name: Gather IOS Facts
        ios_facts:
      - debug: var=ansible_net_version
      - name: Set Facts IOS
        set_fact:
          ios_version: "{{ ansible_net_version }}"

      - name: Create Output file
        file: path=/tmp/test state=directory mode=0755
        delegate_to: 127.0.0.1
        run_once: true

      - name: Run Template
        template:
          src: ios_firmware_check.j2
          dest: /tmp/test/output.txt
        delegate_to: 127.0.0.1
        run_once: true

      - name: Set up keys
        become: yes
        become_method: su
        become_user: different_user
        authorized_key:
          user: different_user
          state: present
          key: "{{ lookup('file', '/home/different_user/.ssh/key_file') }}"
        delegate_to: 127.0.0.1
        run_once: true
      - name: Copy to remote server
        remote_user: different_user
        copy:
          src: /tmp/test/output.txt
          dest: /tmp/test/output.txt
        delegate_to: remote.server.fqdn
        run_once: true

When run, the playbook fails in the Set up keys task trying to access the home directory with the ssh key:

TASK [Set up keys] *************************************************************
task path: /tmp/awx_2206_mz90qvh9/project/IOS/ios_version.yml:23
 [WARNING]: Unable to find '/home/different_user/.ssh/key_file' in expected paths
(use -vvvvv to see paths)
File lookup using None as file
fatal: [host]: FAILED! => {
    "msg": "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'ansible.errors.AnsibleError'>, original message: could not locate file in lookup: /home/different_user/.ssh/key_file"
}

I'm assuming my mistake is somehow related to which user is trying to access the /home/ directory on which device. Is there a better/more elegant/working way of connecting to a different server using an ssh key to move around files? I know one possibility would be to just scp using the shell module, but that always feels a bit hacky.

Solution

(sort of) solved using encrypted variables in hostvars with Ansible vault. How to get there:

Encrypting the passwords:

This needs to be done from any commandline with Ansible installed, for some reason this can't be done in tower/awx

ansible-vault encrypt_string "password"

You'll be prompted for a password to encrypt/decrypt. If you're doing this for Cisco devices, you'll want to encrypt both the ssh and the enable password using this method.

Add encrypted passwords to inventory

For testing, I put it in hostvars for a single switch, should be fine to put it into groupvars and use it on multiple switches as well.

ansible_ssh_pass should be the password to access the switch, ansible_become_pass is the enable password.

---
all:
  children:
    Cisco:
      children:
        switches:
switches:
  hosts:
    HOSTNAME:
      ansible_host: ip-address
      ansible_user: username
      ansible_ssh_pass: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          [encrypted string]
      ansible_connection: network_cli
      ansible_network_os: ios
      ansible_become: yes
      ansible_become_method: enable
      ansible_become_pass: !vault |
          $ANSIBLE_VAULT;1.1;AES256
         [encrypted string]

Adding the vault password to tower/awx

Add a new credential with credential type "Vault" and the password you used earlier to encrypt the strings. Now, all you need to do is add the credential to your job template (the template can have one "normal" credential (machine, network, etc.) and multiple vaults).

The playbook then automagically accesses the vault credential to decrypt the strings in the inventory.

Playbook to get Switch Infos and drop template file on a server

The playbook now looks something like below, and does the following:

Gather Facts on all Switches in Inventory
Write all facts into a .csv using a template, save the file on the ansible host
Copy said file to a different server using a different user

The template is configured with the user able to access the server, the user used to access switches with a password is stored in the inventory as seen above.

---
  - name: Read Switch Infos
    hosts: all
    gather_facts: no
    tasks:
      - name: Create Output file
        file: path=/output/directory state=directory mode=0755
        delegate_to: 127.0.0.1
        run_once: true
      - debug:
          var: network
      
      - name: Gather IOS Facts
        remote_user: username
        ios_facts:
      - debug: var=ansible_net_version
      
      - name: Set Facts IOS
        set_fact:
          ios_version: "{{ ansible_net_version }}"

      - name: Run Template
        template:
          src: ios_firmware_check.csv.j2
          dest: /output/directory/filename.csv
        delegate_to: 127.0.0.1
        run_once: true
      
      - name: Create Destination folder on remote server outside inventory
        remote_user: different_username
        file: path=/destination/directory mode=0755
        delegate_to: remote.server.fqdn
        run_once: true
      
      - name: Copy to remote server outside inventory
        remote_user: different_username
        copy:
          src: /output/directory/filename.csv
          dest: /destination/directory/filename.csv
        delegate_to: remote.server.fqdn
        run_once: true