Ansible Syncronize Keeps failing with Permission denied

I have this simple Ansible setup:

host_vars/testserver.yaml

---
ansible_host: x.x.x.x
ansible_connection: ssh
ansible_port: 22
ansible_ssh_user: ansible
ansible_become: true
ansible_become_user: root
ansible_password: "PASSWORD HERE"
ansible_ssh_pass: "PASSWORD HERE"
ansible_sudo_pass: "PASSWORD HERE"
ansible_become_pass: "PASSWORD HERE"
hostname: "{{ inventory_hostname }}"

sync.yaml

---
- hosts: all
   gather_facts: true

   tasks:
   - name: create a file (works)
     copy:
       dest: "/root/blah"
       content: |
         line 01
         line 02

   - name: copy app-scraper (fails!)
     synchronize:
       src: files/var/opt/app-scraper
       dest: /var/opt/
       rsync_opts:
         - "--exclude=.git"
         - "--exclude=__pycache__"

I keep getting Permission denied (publickey,password). But other ansible builtins like copy, tempalte, unarchive, file, cron, ... work fine which shows user/pass is working. It is just synchronize that fails!!!

fatal: [lmoc1]: FAILED! => {"changed": false, "cmd": "sshpass -d3 /usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' --rsync-path='su -c rsync' -e \"ssh\" --exclude=.git --exclude=__pycache__ --out-format='<<CHANGED>>%i %n%L' /home/leomoon/ansible/files/var/opt/app-scraper [email protected]:/var/opt/", "msg": "Permission denied, please try again.\r\nPermission denied, please try again.\r\[email protected]: Permission denied (publickey,password).\r\nrsync error: unexplained error (code 255) at rsync.c(703) [sender=3.2.3]\n", "rc": 5}

Any help is appreciated. :)

Solution

Here's what's going on. In short, if you are re-assigning a variable, it will not be evaluated with Synchronize.

My setup is like this. Ansible will get the value of {{ ansible_pass }} from secrets.yaml and assigns it to all these variables...

ansible_password: "{{ ansible_pass }}"
ansible_ssh_pass: "{{ ansible_pass }}"
ansible_sudo_pass: "{{ ansible_pass }}"
ansible_become_pass: "{{ ansible_pass }}"

This will work for all ansible modules except Synchronize.

Test case

ansible/
    host_vars/
        testserver.yaml
    secrets.yaml
    sync_test.yaml

host_vars/testserver.yaml

---
ansible_host: x.x.x.x
ansible_connection: ssh
ansible_port: 22
ansible_ssh_user: ansible
ansible_become: true
ansible_become_user: root
ansible_password: "{{ ansible_pass }}"
ansible_ssh_pass: "{{ ansible_pass }}"
ansible_sudo_pass: "{{ ansible_pass }}"
ansible_become_pass: "{{ ansible_pass }}"
hostname: "{{ inventory_hostname }}"

secrets.yaml

This has the ssh password here and it is encrypted. But for our test we can leave it not encrypted.

ansible_pass: PASSWORDHERE

sync_test.yaml

---
- hosts: all
  gather_facts: true
  vars_files:
    - secrets.yaml
  tasks:

  - debug: msg="{{ ansible_pass }} {{ ansible_password }} {{ ansible_ssh_pass }} {{ ansible_sudo_pass }} {{ ansible_become_pass }}"

  - name: create a file (works)
    copy:
      dest: "/tmp/blah"
      content: |
        line 01
        line 02

  - name: copy app-scraper (fails!)
    synchronize:
      src: flask/files/var/opt/app-scraper
      dest: /tmp/
      rsync_opts:
        - "--exclude=.git"
        - "--exclude=__pycache__"

And if you run this you can see the debug message is shwoing the correct password but it will fail at synchronize.

$ ansible-playbook sync_test.yaml -l testserver
Vault password: 

PLAY [all] ******************************************************************

TASK [Gathering Facts] ******************************************************
ok: [testserver]

TASK [debug] ****************************************************************
ok: [testserver] => {
    "msg": "PASSWORDHERE PASSWORDHERE PASSWORDHERE PASSWORDHERE PASSWORDHERE"
}

TASK [create a file (works)] ************************************************
ok: [testserver]

TASK [copy app-scraper (fails!)] ********************************************
fatal: [testserver]: FAILED! => {"changed": false, "cmd": "sshpass -d3 /usr/bin/rsync --delay-updates -F --compress --archive --rsh='/usr/bin/ssh -S none -o Port=22 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' --rsync-path='sudo -u root rsync' --exclude=.git --exclude=__pycache__ --out-format='<<CHANGED>>%i %n%L' /home/leomoon/ansible/flask/files/var/opt/app-scraper [email protected]:/tmp/", "msg": "Warning: Permanently added 'x.x.x.x' (ED25519) to the list of known hosts.\r\nPermission denied, please try again.\r\nPermission denied, please try again.\r\[email protected]: Permission denied (publickey,password).\r\nrsync error: unexplained error (code 255) at rsync.c(713) [sender=3.2.7]\n", "rc": 5}

PLAY RECAP ******************************************************************
testserver                  : ok=3    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Temporary fix

Ansible is not passing the password to Synchronize and this is a bug but the temporary fix is to set_fact those variables again! This was tested on ansible version 2.14.1:

$ ansible --version
ansible [core 2.14.1]
  python version = 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.0.3
  libyaml = True

So the sync_test.yaml should be changed to this:

---
- hosts: all
  gather_facts: true
  vars_files:
    - secrets.yaml
  tasks:

  - debug: msg="{{ ansible_pass }} {{ ansible_password }} {{ ansible_ssh_pass }} {{ ansible_sudo_pass }} {{ ansible_become_pass }}"

  - name: create a file (works)
    copy:
      dest: "/tmp/blah"
      content: |
        line 01
        line 02

  - name: temp fix for synchronize to force evaluate variables!
    set_fact:
      ansible_password: "{{ ansible_pass }}"
      ansible_ssh_pass: "{{ ansible_pass }}"
      ansible_sudo_pass: "{{ ansible_pass }}"
      ansible_become_pass: "{{ ansible_pass }}"

  - name: rsync app-scraper (fixed!)
    synchronize:
      src: flask/files/var/opt/app-scraper
      dest: /tmp/
      rsync_opts:
        - "--exclude=.git"
        - "--exclude=__pycache__"

And this will work!