GitLab CI runner with SSH ProxyJump

Question

I have the following settings in my /etc/ssh/ssh_config file:

Host serverA
  User idA
Host serverB
  User idB
  ProxyJump serverA

I’ve also copied the public keys, so if I locally run ssh serverB I’m correctly connected to serverB as idB through serverA.

Now, here’s my runner configuration in /etc/gitlab-runner/config.toml:

[[runners]] 
name = "ssh-runner-1"                                                                
url = "http://my-cicd-server"                                    
token = "xxxxxxxxxxxxxxxx"                                                 
executor = "ssh"                                                               
[runners.custom_build_dir]                                                     
[runners.cache]                                                                
  [runners.cache.s3]                                                           
  [runners.cache.gcs]                                                          
  [runners.cache.azure]                                                        
[runners.ssh]                                                                  
  user = "idB"                                                    
  host = "serverB"                                                    
  identity_file = "/home/gitlab-runner/.ssh/id_ed25519"

When I run a CI/CD job on this runner I get a « connection refused » error:

ERROR: Preparation failed: ssh command Connect() error: ssh Dial() error: dial tcp xxx.xxx.xxx.xxx:22: connect: connection refused

I conclude that the ProxyJump configuration is not applied, and since the machine with the runner can’t directly connect to serverB, I get denied access. How can I configure the runner to apply the proxy jump configuration?

Answer 1

The GitLab runner uses a Go-based SSH client. It does not respect your system SSH configuration and does not have the same configurability as the standard ssh (usually OpenSSH) packages you typically find installed in operating system distributions or similar packages.

The Go client does not support the ProxyJump configuration.

Your best bet would probably be to configure a tunneled connection where your entrypoint does not require SSH configuration options that are not supported.

Local port forwarding

One way might be to open a local port-forwarding tunnel, then in your GitLab configuration, specify the host as localhost and port as the forwarded port.

For example:

1. Open the tunnel -- local port 2222 forwards to port 22 on ServerB via ssh connection through ServerA

ssh -L 2222:ServerB:22 -N ServerA

2. Configure runner to use the tunnel:

...
[runners.ssh]                                                                  
  host = "localhost"                                                    
  port = 2222
...

With this approach, you may have to write some automation on your server to restore the tunnel connection in the event it is broken. How you might do this depends on your operating system and preferred service manager. Or use a tool like autossh

This is basically how the ProxyJump configuration works under the hood.

IP/Port forwarding system

A similar approach would be to have your jump system automatically forward connections to the desired destination. This might be something like a software firewall rule (e.g. using iptables routing rules). That way the forwarding occurs transparently. Then simply tell the runner to target ServerA and the traffic will be transparently moved to ServerB.

This approach is more reliable, since you won't have to do anything to keep the tunnel alive if it ever drops. However, the configuration is much more complex and requires a static IP for ServerB.

For example, on ServerA, assuming the IP of ServerB is 10.10.10.10 the following iptables configuration could be used:

iptables -t nat -A PREROUTING -p tcp --dport 2222 -j DNAT --to-destination 10.10.10.10:22
iptables -t nat -A POSTROUTING -j MASQUERADE

reference.

Then the GitLab runner configuration:

...
[runners.ssh]                                                                  
  host = "ServerA"                                                    
  port = 2222
...

Lastly, it may also be useful to know that disable_strict_host_key_checking is an undocumented configuration option for the runner as well, in the event you need this.