I have a staging server and a production server, and I run identical Bitbucket Pipelines, where I send some commands over SSH. Unfortunately, my pipeline for the production always fails with:
Host key verification failed.
I've tried everything, folder permissions, recreating the keys, nothing works.
Finally with adding -v
to my ssh
call, I think I'm a step closer, but still lost.
On my staging server, I see something like this:
debug1: Host '$STAGING_SERVER' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug1: ssh_rsa_verify: signature correct
and the rest of the build follows flawlessly.
On my production server, however, I see the following:
debug1: Host '$PRODUCTION_SERVER' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:5
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: permanently_drop_suid: 0
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Host key verification failed.
So it would look like the key is found on my production server, but for some reason on the production server read_passphrase
is being called. I've just created a new id_rsa and id_rsa.pub key, with no password, so why the heck is my production server trying to call read_passphrase
? My ssh_config
and sshd_config
on both servers are identical - checked via diff
.
Another way of looking at it is that ssh_rsa_verify
is called immediately on the staging server, while on the production server read_passphrase
is called.
Any help here would be greatly appreciated, this is driving me crazy!
Hallelujah! Solved! 🥳
Hours wasted for the simplest reason...
I noticed in the full output of the ssh -v
on production that Bitbucket was printing out something like this:
debug1: Connecting to $PRODUCTION_SERVER [12.345.567.890] port 22.
where as the staging output was:
debug1: Connecting to $STAGING_SERVER [$STAGING_SERVER] port 22.
Meaning the static IP was the exact value of that repository variable. (Bitbucket parses secret logs out, which is why they appear this way).
I realized I had set the repository variable PRODUCTION_SERVER
incorrectly to the alias for the IP address, (i.e. myserver.com
) when it should be the IP address exactly. Changing that value in my repository variables to the IP address fixed the issue! Apparently, using the alias name isn't a perfect enough match for SSH to be satisfied.