I'm told this is the most secure method, but I'm not able to piece together a working solution based on the scattered and mostly outdated info I'm finding out there.
Using AWS CDK v2 to deploy the bastion, the RDS instance, so I can then connect using ssh through a terminal or DBeaver.
I tried following this article, and here's where I settled:
export class SshBastionRdsStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
const vpc = new Vpc(this, "SshBastionRdsVpc");
const selfRefSg = new SecurityGroup(this, "sg-SshBastionRdsSelfRef", {
vpc,
securityGroupName: "SshBastionRdsSelfRefSecurityGroup",
description: "Self-referencing for port 5432",
});
selfRefSg.addIngressRule(selfRefSg, Port.tcp(5432), "Self-referencing for port 5432");
const secret = new Secret(this, "SshBastionRdsUserSecret", {
secretName: "pg-test-master-user-secret",
description: "Database master user credentials",
generateSecretString: {
secretStringTemplate: JSON.stringify({ username: "postgres" }),
generateStringKey: "password",
passwordLength: 16,
excludePunctuation: true
},
});
const username = secret.secretValueFromJson("username").unsafeUnwrap().toString();
const password = secret.secretValueFromJson("password");
const parameterGroup = new ParameterGroup(this, "SshBastionRdsParameterGroup", { engine });
parameterGroup.addParameter("rds.force_ssl", "0"); //negates need for "ssl" prop on pg Client
parameterGroup.addParameter("password_encryption", "md5"); //allows non-SCRAM users to auth
const rdsInstance = new DatabaseInstance(this, "SshBastionRdsDb", {
vpc,
parameterGroup,
credentials: { username, password },
engine,
instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.MICRO),
securityGroups: [selfRefSg],
backupRetention: Duration.days(0),
deleteAutomatedBackups: true,
removalPolicy: RemovalPolicy.DESTROY,
});
const bastionHost = new BastionHostLinux(this, "SshBastionRdsBastionHost", {
instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.NANO),
machineImage: MachineImage.latestAmazonLinux2(),
subnetSelection: { subnetType: SubnetType.PUBLIC },
securityGroup: selfRefSg,
vpc
});
bastionHost.allowSshAccessFrom(Peer.anyIpv4());
rdsInstance.connections.allowFrom(
bastionHost.connections,
Port.tcp(rdsInstance.instanceEndpoint.port),
"Bastion host connection"
);
new CfnOutput(this, "SshBastionRdsBastionHostOutput", {
value: bastionHost.instanceId
});
}
}
Then I generated a keypair:
ssh-keygen -t rsa -m pem
Change permissions on the private key:
sudo chmod 400 ~/.ssh/cdk-ssh-bastion-rds
Added public key to bastion host:
aws ec2-instance-connect send-ssh-public-key \
--instance-id i-xxxxxxxxxxxxxxx \
--instance-os-user ec2-user \
--ssh-public-key file:///Users/me/.ssh/cdk-ssh-bastion-rds.pub
I can successfully connect to the bastion ec2 instance:
ssh -i ~/.ssh/cdk-ssh-bastion-rds -l ec2-user XXX.XX.XXX.XX
...which gives me:
Last login: Sun Mar 3 02:17:11 2024 from xxxx.xxxx.xxxx.xxxx
, #_
~\_ ####_ Amazon Linux 2
~~ \_#####\
~~ \###| AL2 End of Life is 2025-06-30.
~~ \#/ ___
~~ V~' '->
~~~ / A newer version of Amazon Linux is available!
~~._. _/
_/ _/ Amazon Linux 2023, GA and supported until 2028-03-15.
_/m/' https://aws.amazon.com/linux/amazon-linux-2023/
[ec2-user@ip-10-0-2-69 ~]$
Weird thing is, I can't do this twice. I get the following if I exit and try again:
ec2-user@XXX.XX.XXX.XX: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
If I aws ec2-instance-connect send-ssh-public-key once more and try again...it connects again. What's that all about? Is it not storing the public key between sessions?
Anyhow...moving on.
If I try to hit RDS through the bastion:
ssh -i ~/.ssh/cdk-ssh-bastion-rds -f -N -L 22:sshbastionrdsstack-sshbastionrdsxxxxxxxxx-xxxxxxxxxxxxxx.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432 ec2-user@XXX.XX.XXX.XX -v
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/me/.ssh/config
debug1: /Users/me/.ssh/config line 2: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to XXX.XX.XXX.XX [XXX.XX.XXX.XX] port 22.
debug1: Connection established.
debug1: identity file /Users/me/.ssh/cdk-ssh-bastion-rds type 0
debug1: identity file /Users/me/.ssh/cdk-ssh-bastion-rds-cert type -1
debug1: identity file /Users/me/.ssh/id_rsa_personal type 0
debug1: identity file /Users/me/.ssh/id_rsa_personal-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: compat_banner: match: OpenSSH_7.4 pat OpenSSH_7.4* compat 0x04000006
debug1: Authenticating to XXX.XX.XXX.XX:22 as 'ec2-user'
debug1: load_hostkeys: fopen /Users/me/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
debug1: load_hostkeys: fopen /Users/me/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'XXX.XX.XXX.XX' is known and matches the ED25519 host key.
debug1: Found key in /Users/me/.ssh/known_hosts:26
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: get_agent_identities: bound agent to hostkey
debug1: get_agent_identities: agent returned 3 keys
debug1: Will attempt key: /Users/me/.ssh/id_rsa_personal RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx explicit agent
debug1: Will attempt key: /Users/me/.ssh/cdk-ssh-bastion-rds RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx explicit agent
debug1: Will attempt key: /Users/me/.ssh/cdk-ssh-bastion-rds RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx agent
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/me/.ssh/id_rsa_personal RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx explicit agent
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /Users/me/.ssh/cdk-ssh-bastion-rds RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx explicit agent
debug1: Server accepts key: /Users/me/.ssh/cdk-ssh-bastion-rds RSA SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx explicit agent
Authenticated to XXX.XX.XXX.XX ([XXX.XX.XXX.XX]:22) using "publickey".
debug1: Local connections to LOCALHOST:22 forwarded to remote address sshbastionrdsstack-sshbastionrdsxxxxxxxxx-xxxxxxxxxxxxxx.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432
debug1: Local forwarding listening on ::1 port 22.
bind [::1]:22: Permission denied
debug1: Local forwarding listening on 127.0.0.1 port 22.
bind [127.0.0.1]:22: Permission denied
channel_setup_fwd_listener_tcpip: cannot listen to port: 22
Could not request local forwarding.
debug1: Requesting no-more-sessions@openssh.com
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: filesystem
(base) me@VGJ-MBP-M2X cdk-ssh-bastion-rds % debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /Users/me/.ssh/known_hosts for XXX.XX.XXX.XX / (none)
debug1: client_input_hostkeys: searching /Users/me/.ssh/known_hosts2 for XXX.XX.XXX.XX / (none)
debug1: client_input_hostkeys: hostkeys file /Users/me/.ssh/known_hosts2 does not exist
debug1: client_global_hostkeys_private_confirm: server used untrusted RSA signature algorithm ssh-rsa for key 0, disregarding
debug1: update_known_hosts: known hosts file /Users/me/.ssh/known_hosts2 does not exist
It appears I'm connected and authenticated just fine. However, I can't connect with psql at this point. It times out:
(base) me@VGJ-MBP-M2X ~ % psql -U postgres -h sshbastionrdsstack-sshbastionrdsdbxxxxxxxx-xxxxxxxxxxxxx.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com -p 5432 -d postgres
psql: error: connection to server at "sshbastionrdsstack-sshbastionrdsdbxxxxxxxx-xxxxxxxxxxxxx.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com" (XX.X.XXX.XXX), port 5432 failed: Operation timed out
Is the server running on that host and accepting TCP/IP connections?
I would guess port 22 on your local is either already bound to SSH, or you cant bind that port because its below 1024 therefore its a privileged port (so it would work(if not in use on your local) only when running SSH with sudo/administrator privileges - but this is not good practice and not necessary).
I would suggest you change port 22 in your -L 22:sshbastionrdsstack-sshbastionrdsxxxxxxxxx-xxxxxxxxxxxxxx.xxxxxxxxxxxxx.us-east-1.rds.amazonaws.com:5432 argument using a port above above 1024(5432 might be a good choice if your not also running a local DB instance, but it doesn't matter what port really so long as its above 1024 and not in use). Once SSH is connected, you will then be able to make connections to localhost:5432 (or whatever port you use for the 1st number passed to -L argument). The connection will be forwarded from localhost:[your port] -> over SSH port 22 to the instance -> then from the bastion to the destination using the port specified in the 3rd argument of the -L argument.
You might also need to add a security group to allow the bastion host to talk with RDS on 5432. The recommended way to do this is with a "self referencing security group" - that is a security group that allows 5432 from it self only. Then adding that security group to both the RDS and the bastion instances. Ive not tested it but there is a CDK gist on Github here.
As for your SSH Key issues, thats expected:
Pushes an SSH public key to the specified EC2 instance for use by the specified user. The key remains for 60 seconds. For more information, see Connect to your Linux instance using EC2 Instance Connect in the Amazon EC2 User Guide .
2 options come to mind:
echo "[Paste your Pub Key here]" >> ~/.ssh/authorizied_keys && chmod 400 ~/.ssh/authorizied_keys to persist your key.