Search code examples
sshwindows-subsystem-for-linuxparamikoopenssh

Paramiko sftp fails SSH to windows with WSL being default shell of OpenSSH: SSHException: EOF during negotiation


On server side, I set WSL to be default shell in OpenSSH using:

New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\wsl.exe" -PropertyType String -Force

(simply following official instructions of OpenSSH)

However, after this modification, Paramiko sftp fails to instantiate a connection with SSHException: EOF during negotiation

Before that change, all looks good. Additionally, SSH connects normally, its just sftp that fails. Simliar thing happend when I do things from terminal (without Python invloved).

Python code:

import paramiko
ssh = paramiko.SSHClient()
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(hostname='alex-surface', username='alex')
sftp = ssh.open_sftp()

Error at sftp:

---------------------------------------------------------------------------
EOFError                                  Traceback (most recent call last)
File ~\venvs\ve\lib\site-packages\paramiko\sftp_client.py:130, in SFTPClient.__init__(self, sock)
    129 try:
--> 130     server_version = self._send_version()
    131 except EOFError:

File ~\venvs\ve\lib\site-packages\paramiko\sftp.py:134, in BaseSFTP._send_version(self)
    133 self._send_packet(CMD_INIT, struct.pack(">I", _VERSION))
--> 134 t, data = self._read_packet()
    135 if t != CMD_VERSION:

File ~\venvs\ve\lib\site-packages\paramiko\sftp.py:201, in BaseSFTP._read_packet(self)
    200 def _read_packet(self):
--> 201     x = self._read_all(4)
    202     # most sftp servers won't accept packets larger than about 32k, so
    203     # anything with the high byte set (> 16MB) is just garbage.

File ~\venvs\ve\lib\site-packages\paramiko\sftp.py:188, in BaseSFTP._read_all(self, n)
    187 if len(x) == 0:
--> 188     raise EOFError()
    189 out += x

EOFError:

During handling of the above exception, another exception occurred:

SSHException                              Traceback (most recent call last)
Input In [7], in <cell line: 6>()
      4 ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
      5 ssh.connect(hostname='alex-surface', username='alex')
----> 6 sftp = ssh.open_sftp()

File ~\venvs\ve\lib\site-packages\paramiko\client.py:558, in SSHClient.open_sftp(self)
    552 def open_sftp(self):
    553     """
    554     Open an SFTP session on the SSH server.
    555
    556     :return: a new `.SFTPClient` session object
    557     """
--> 558     return self._transport.open_sftp_client()

File ~\venvs\ve\lib\site-packages\paramiko\transport.py:1142, in Transport.open_sftp_client(self)
   1132 def open_sftp_client(self):
   1133     """
   1134     Create an SFTP client channel from an open transport.  On success, an
   1135     SFTP session will be opened with the remote host, and a new
   (...)
   1140         this transport
   1141     """
-> 1142     return SFTPClient.from_transport(self)

File ~\venvs\ve\lib\site-packages\paramiko\sftp_client.py:170, in SFTPClient.from_transport(cls, t, window_size, max_packet_size)
    168     return None
    169 chan.invoke_subsystem("sftp")
--> 170 return cls(chan)

File ~\venvs\ve\lib\site-packages\paramiko\sftp_client.py:132, in SFTPClient.__init__(self, sock)
    130     server_version = self._send_version()
    131 except EOFError:
--> 132     raise SSHException("EOF during negotiation")
    133 self._log(
    134     INFO,
    135     "Opened sftp connection (server version {})".format(
    136         server_version
    137     ),
    138 )

SSHException: EOF during negotiation

Solution

  • Short answer

    Use the Windows SSH server as a jump host to the WSL SSH server.

    More details

    While using WSL as your shell for the Windows SSH server works for simple interactions, it starts to have issues with more complex cases like:

    • sftp
    • sshfs
    • scp
    • Ansible
    • Tunneling/port-forwarding uses
    • And more

    However, it's possible (and, IMHO, preferable) to get "real" SSH access to the WSL instance (without port forwarding) by using the Windows SSH server as a jump host to the WSL SSH server.

    Advantages of this method:

    • No port forwarding required
    • Minimal firewall work needed since it relies on Windows services
    • Works even if you have multiple WSL distributions installed
    • Works with sshfs, scp, sftp, Ansible, and any app or service that requires a real SSH connection.

    The summary is that:

    • We rely on WSL2's ability to forward "localhost" connections on Windows to WSL (a.k.a. localhost forwarding).
    • We use Windows OpenSSH server as a JumpHost to the WSL2 OpenSSH server.

    You've already done the initial setup for this, but I'll repeat it here for other potential readers:

    Part 1 - Configure Windows OpenSSH Server

    1. For you, the first step will be to remove the registry key that you created that sets WSL as your OpenSSH shell. Most other readers won't have to worry about this.

    You've done most of the following steps already, so scan through this, but mostly you'll skip to "Part 2" below. The next few steps in this section are for users that are configuring this for the first time.

    1. Start by enabling the Windows OpenSSH server on port 22.

      (Especially for future readers) I recommend simply following the Microsoft docs for the latest information, but I'm copying the relevant commands in here as well. From PowerShell:

      # Sounds like you had the Client already installed, at least
      Add-WindowsCapability -Online -Name OpenSSH.Client~~~~0.0.1.0 
      Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0
      
      Start-Service sshd
      Set-Service -Name sshd -StartupType 'Automatic'
      
      # Confirm the Firewall rule is configured. It should be created automatically by setup. Run the following to verify
      if (!(Get-NetFirewallRule -Name "OpenSSH-Server-In-TCP" -ErrorAction SilentlyContinue | Select-Object Name, Enabled)) {
          Write-Output "Firewall Rule 'OpenSSH-Server-In-TCP' does not exist, creating it..."
          New-NetFirewallRule -Name 'OpenSSH-Server-In-TCP' -DisplayName 'OpenSSH Server (sshd)' -Enabled True -Direction Inbound -Protocol TCP -Action Allow -LocalPort 22
      } else {
          Write-Output "Firewall rule 'OpenSSH-Server-In-TCP' has been created and exists."
      }
      
    2. (Optional, but recommended) If you are an Administrator on your Windows installation -- Edit C:\Program Data\ssh\sshd_config and comment out the following lines:

      #Match Group administrators
      #       AuthorizedKeysFile __PROGRAMDATA__/ssh/administrators_authorized_keys
      

      By default, Windows OpenSSH looks for C:\Program Data\ssh\administrators_authorized_keys (see this question). By commenting these lines out, you will use your %userprofile%\.ssh\authorized_keys instead.

    3. (Optional) While you are editing that, you may want to set:

      PasswordAuthentication no
      
    4. Add your public key to %userprofile%\.ssh\authorized_keys. Make sure that permissions are restrictive per this answer (or related answers).

    5. Confirm that you can log to Windows OpenSSH from WSL using your key:

      ssh -i <path_to_private_key> "${HOSTNAME}.local"
      

    At this point, you really have terminal access to WSL2 through Windows OpenSSH already.

    From any machine on the network:

    ssh -t <windows_host_or_ip> wsl
    

    This will simply run the wsl command after the connection to Windows.

    And that works for shell access, just as when you had changed the default shell. But now we get into the root of the answer for your particular use-case:

    Part 2 - Configure OpenSSH in WSL

    1. Unless I missed it, you don't mention which distribution you are using in WSL. Of course, the SSH server configuration steps will vary depending on your distribution. I'll assume Ubuntu here, since it's the most common (and default) distribution on WSL.

    2. In WSL, sudo -e /etc/ssh/sshd_config. Uncomment the Port line and change it to something other than 22. A good option is 2222. Note that if you use multiple WSL distributions, each one will need a different port number.

      Change any other options you need to here. Ubuntu has sensible defaults, but review as needed.

    3. If this is your first time running the SSH server in Ubuntu, generate the Host keys with:

      sudo dpkg-reconfigure openssh-server
      
    4. Start the SSH server with:

      sudo service ssh start
      
    5. Copy over your public key to ~/.ssh/authorized_keys (already done for you) and make sure your permissions are correct (as mentioned above).

    6. Add your private key to ssh-agent via:

      eval $(ssh-agent) # under Linux
      ssh-add <path_to_key
      

      Side-note 1: Windows also supports ssh-add. Just make sure the "OpenSSH Authentication Service" is running).

      Side-note 2: I personally prefer using Keychain (written by the founder of Gentoo) along with the Fish Shell as I mention here. That combination allows the ssh-agent for all running shells to be kept in sync.


    At this point, you can use the Windows host as your JumpHost like so:

    ssh -J <windows_host_or_ip> -p 2222 localhost
    

    Typically, you can use mDNS to obtain the correct Windows host IP via:

    ssh -J $(hostname).local -p 2222 localhost
    

    This connects to the Windows OpenSSH server (on port 22) which then turns around and connects to localhost:2222, which is your WSL2 instance.

    As mentioned, this will now work for scp, sshfs, etc.

    For instance:

    sftp -J $(hostname).local -P 2222 localhost
    

    Optional for non-Paramiko usage

    This technique has one "side-effect", in that localhost is stored as the same "known host" regardless of which port or jump host you use to connect. So if you do connect to multiple WSL instances in this way, ssh will start complaining about potential man-in-the-middle issues.

    The best way to avoid this (and simplify things in general) is to create a Host entry in ~/.ssh/config. Let's say your Windows host name is bubblegum and your WSL distro is ubuntu. Add the following to ~/.ssh/config:

    Host bubblegum_ubuntu  # Can be whatever you want
    Hostname localhost
    User <username> # If needed
    Port 2222
    ProxyJump bubblegum
    UserKnownHostsFile ~/.ssh/known_hosts_bubblegum_ubuntu
    

    That will redirect the known_host entry to a file that is only used for that particular host.

    It also means that you can now:

    ssh bubblegum_ubuntu
    scp bubblegum_ubuntu:/home/username/filename .
    sftp bubblegum_ubuntu
    sshfs bubblegum_ubuntu:/ /mountpoint
    

    ... without the need to manually specify the jump host each time.

    Using jump hosts in Paramiko

    This appears to be covered in this SO answer.

    I haven't tested this part myself (yet), but you may be able to do it more quickly than I, so I'll go ahead and post as-is for now.

    As an aside, I will say that the Fabric solution (which is built on top of Paramiko) appears to be much simpler.

    Note that neither Paramiko nor Fabric can make use of the ~/.ssh/config for configuring the jump host using the config example above. While Paramiko can parse the config for some keywords, ProxyJump isn't one of them.

    However, that doc does refer to ProxyCommand which sounds like it may be an alternative method.