I'm having a really weird problem where when I connect to an SSH server (that has a pretty standard default configuration, including a default MaxAuthTries of 6) using Net::SSH, the connection fails after 3 authentication attempts with a disconnected: Too many authentication failures (2) (Net::SSH::Disconnect) exception while the OpenSSH command line client connects fine.
The problem is that my SSH Agent already knows about 4 identities and offers them automatically - annoyingly before the identity that is configured in the SSH client configuration file for that server - and so the OpenSSH client always succeeds to connect and the Net::SSH client always fails to connect. I can get the Net::SSH client to connect by disabling the SSH Agent.
$ ssh-add -l
2048 SHA256:fMBMXD1mpUpM/vmVcjaXDvdIvPy4GCCdG5lc8ga1cLU /home/odeda/.ssh/id_rsa (RSA)
3072 SHA256:nfsiFP9APVl6oOK5htpldBmknVTDt1lCLvuHpTaaAao odeda@vesho (RSA)
2048 SHA256:t3+SY5Ru+tJ3r/HE7qIW4fnXswQ16btCiIadVGa2XPM odeda@jior (RSA)
1024 SHA256:EKBzRjDE6twnRLlIT3IHEfu08tVXCSK1henMHB+s+p8 (DSA)
$ grep example.com ~/.ssh/config -A1
Host *.example.com
IdentityFile ~/.ssh/my_example_identity.pem
$ bundle exec ruby -rnet/ssh -e 'Net::SSH.start("server.example.com","ubuntu",{
host_name:"3.456.789.12",
verify_host_key: :never,
user_known_hosts_file: "/dev/null",
non_interactive: true}) do |ssh|
puts "Connected";
puts ssh.exec! "echo Hello";
ssh.loop(3);
end'
~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:198:in `block in poll_message':
disconnected: Too many authentication failures (2) (Net::SSH::Disconnect)
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:190:in `loop'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:190:in `poll_message'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/transport/session.rb:175:in `next_message'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:103:in `block in next_message'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:102:in `loop'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:102:in `next_message'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:50:in `authenticate_with_alg'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:100:in `block in authenticate_with'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:97:in `each'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:97:in `authenticate_with'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:19:in `block in authenticate'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/key_manager.rb:132:in `block in each_identity'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/key_manager.rb:124:in `each'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/key_manager.rb:124:in `each_identity'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/methods/publickey.rb:18:in `authenticate'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:88:in `block in authenticate'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:72:in `each'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh/authentication/session.rb:72:in `authenticate'
from ~/my-project/.vendor/bundle/ruby/3.0.0/gems/net-ssh-7.2.3/lib/net/ssh.rb:262:in `start'
from -e:1:in `<main>'
$ ssh -oHostname=3.456.789.12 -l ubuntu server.example.com -- echo Hello
Hello
$ SSH_AUTH_SOCK= bundle exec ruby -rnet/ssh -e 'Net::SSH.start("server.example.com","ubuntu",{
host_name:"3.456.789.12",
verify_host_key: :never,
user_known_hosts_file: "/dev/null",
non_interactive: true}) do |ssh|
puts "Connected";
puts ssh.exec! "echo Hello";
ssh.loop(3);
end'
Connected
Hello
It gets weirder if we look at the server logs:
This is the OpenSSH client connection:
Jul 03 07:50:44 172-24-51-231.us-east-2.example.com sshd[42437]:
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:fMBMXD1mpUpM/vmVcjaXDvdIvPy4GCCdG5lc8ga1cLU failed, status 22
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:nfsiFP9APVl6oOK5htpldBmknVTDt1lCLvuHpTaaAao failed, status 22
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:t3+SY5Ru+tJ3r/HE7qIW4fnXswQ16btCiIadVGa2XPM failed, status 22
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
userauth_pubkey: key type ssh-dss not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
Accepted publickey for ubuntu from 213.546.879.21 port 47034 ssh2: RSA SHA256:6Azd3HzkKpBLJ+zsolTOpksL4XX8aNdGWtprUwRvsoY
Jul 03 07:50:45 172-24-51-231.us-east-2.example.com sshd[42437]:
pam_unix(sshd:session): session opened for user ubuntu(uid=1000) by (uid=0)
And this is the Net::SSH connection:
Jul 03 07:50:18 172-24-51-231.us-east-2.example.com sshd[42392]:
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:fMBMXD1mpUpM/vmVcjaXDvdIvPy4GCCdG5lc8ga1cLU failed, status 22
Jul 03 07:50:18 172-24-51-231.us-east-2.example.com sshd[42392]:
userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:18 172-24-51-231.us-east-2.example.com sshd[42392]:
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:nfsiFP9APVl6oOK5htpldBmknVTDt1lCLvuHpTaaAao failed, status 22
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
AuthorizedKeysCommand /usr/share/ec2-instance-connect/eic_run_authorized_keys ubuntu SHA256:t3+SY5Ru+tJ3r/HE7qIW4fnXswQ16btCiIadVGa2XPM failed, status 22
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
error: maximum authentication attempts exceeded for ubuntu from 213.546.879.21 port 56414 ssh2 [preauth]
Jul 03 07:50:19 172-24-51-231.us-east-2.example.com sshd[42392]:
Disconnecting authenticating user ubuntu 213.546.879.21 port 56414: Too many authentication failures [preauth]
So we can see that the 4th key I have in my agent - which is a DSA key that is no longer supported by default on modern servers - is sent by the OpenSSH client and rejected because of PubkeyAcceptedAlgorithms and then the 5th public key (from the config file) is accepted. In the case of the Net::SSH client, the 3 RSA keys from the agent - same keys as before - are rejected by PubkeyAcceptedAlgorithms and the fourth isn't sent - instead we get error: maximum authentication attempts exceeded after only 3 tries.
I tried to find a configuration option in Net::SSH (or OpenSSH client configuration) that maybe controls some kind of feature (that I don't know exists) where the client tells the server how many authentication attempts to allow - but of course I couldn't find such because it makes no sense.
I of course tries to set the Net::SSH option number_of_password_prompts to something larger than the default of 3 and of course it didn't do anything because that option is for password prompts, which we are not doing at all.
Any idea will be appreciated, TIA!
The problem is a bug in Net::SSH - issue 886: for some reason Net::SSH can't figure out which key algorithm to use for RSA keys and is by default configured to try both rsa-sha2-256 (a SHA2 variation) and ssh-rsa (SHA-1 which is now rejected by default SSH server configurations), one after another.
As a result, Net::SSH will attempt to authenticate each RSA public key twice - exhausting the default 6 attempts after only 3 RSA keys.
The current workaround is to use the (undocumented) Net::SSH#start option pubkey_algorithms and set it to ["rsa-sha2-256"] (or some other subset of the default algorithm list, if you intend to support certificate based authentication) - this will prevent Net::SSH from using RSA keys with SHA-1 signatures, breaking these types of keys, if you plan to use them.