I am trying to set up an EC2 instance VM on AWS behind a bastion. Ansible performance is very slow but bearable for most operations but file copy operations simply hang and I can't really see what the issue is. The role and playbook work fine connected to a public facing EC2 instance. The copy stanza is:
- name: Copy all files from local dir to remote dir
copy:
src: files/files_to_host/
dest: /home/ec2-user
owner: ec2-user
group: ec2-user
mode: 0644
There are 22 files totalling 22MB with some subdirectories in the src location. I've tried the following ansible.cfg based on Jeff Geerling's post and this Stack Overflow post but it hasn't helped (tried with various lines commented/uncommented):
[ssh_connection]
scp_if_ssh = true
; ssh_args = -o ServerAliveInterval=30
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q myuser@bastion"'
and with it when I run
ansible-playbook -vvvv -i 10.0.129.157, -u ec2-user my-playbook.yml
I get the following error (formatted for legibility):
TASK [Gathering Facts] *********************************************************************************************************************************************************
task path: /Users/myuser/ansible/my-playbook.yml:5
<10.0.129.157> ESTABLISH SSH CONNECTION FOR USER: ec2-user
<10.0.129.157> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ec2-user"' -o ConnectTimeout=10 -o 'ControlPath="/Users/myuser/.ansible/cp/0cdf5a0bfd"' 10.0.129.157 '/bin/sh -c '"'"'echo ~ec2-user && sleep 0'"'"''
<10.0.129.157> (255, b'', b'OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /Users/myuser/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug2: resolve_canonicalize: hostname 10.0.129.157 is address
debug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/Users/myuser/.ssh/known_hosts\'
debug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/Users/myuser/.ssh/known_hosts2\'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/myuser/.ansible/cp/0cdf5a0bfd" does not exist
debug3: ssh_connect_direct: entering
debug1: Connecting to 10.0.129.157 [10.0.129.157] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug2: fd 3 setting O_NONBLOCK
debug1: connect to address 10.0.129.157 port 22: Operation timed out
ssh: connect to host 10.0.129.157 port 22: Operation timed out
')
fatal: [10.0.129.157]: UNREACHABLE! => {
"changed": false,
"msg": "Failed to connect to the host via ssh: OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /Users/myuser/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug2: resolve_canonicalize: hostname 10.0.129.157 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/myuser/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/myuser/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: auto-mux: Trying existing master
debug1: Control socket \"/Users/myuser/.ansible/cp/0cdf5a0bfd\" does not exist
debug3: ssh_connect_direct: entering
debug1: Connecting to 10.0.129.157 [10.0.129.157] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug2: fd 3 setting O_NONBLOCK
debug1: connect to address 10.0.129.157 port 22: Operation timed out
ssh: connect to host 10.0.129.157 port 22: Operation timed out",
"unreachable": true
}
If I use sshuttle as recommended at https://blog.keyboardinterrupt.com/ansible-jumphost/ to open a tunnel to the bastion I can perform remote operations ok, but file copy from local to remote gives errors like
<10.0.129.157> (0, b'/home/ec2-user\n', b"OpenSSH_8.6p1, LibreSSL 3.3.5
debug1: Reading configuration data /Users/myuser/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug2: resolve_canonicalize: hostname 10.0.129.157 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/myuser/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/myuser/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 17624
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 0
")
I am able to use ssh -J and sftp -J successfully and speedily and I am able to access remote host via ssh over sshuttle. I have already added these two lines to the bastion /etc/ssh/sshd_config
MaxStartups 50:30:80
MaxSessions 50
What do I need to do to get Ansible to work correctly via bastion?
Whilst I still don't really know why my previous attempts partially worked rather than completely failing or completely succeeding, I have found a solution that I am pleased with that does not require a central static config file (i.e. no ansible.cfg or ini file at all) or any additional utilities:
ansible-playbook -i 10.0.130.90, -u ec2-user myplaybook.yml --ssh-common-args "-J myuser@bastion"
This presumes ssh-keys for both hosts are added to ssh, e.g. with ssh-add