Search code examples
powershellcharacter-encodingdrone.io

powershell ignore character escaping with output from a different process


I have a process running powershell commands with input from a config file. (In casu it's a drone CI executor on Windows with a drone YML file). This process sets an environment variable but I think this environment variable contains backticks which are interpreted as escape characters by powershell.

Since it's an OpenSSH key, this corrupts the key. Is there a way to treat the contents of the environment variable entirely as raw? Either during setting or reading (because since I don't know the key I do not know where it happens).

Drone CI config:

  - name: copy to archive on remote host
    commands:
      - Write-Output $env:SSH_KEY > ssh_key
      - scp -i ssh_key file.zip user@host:/path/to/file.zip
      - rm ssh_key
    environment:
      SSH_KEY:
        from_secret: ssh_key

I know it's a functional key because it works on my linux containers. But on Windows the format is deemed invalid.


Solution

  • The problem is the behavior of >, the redirection operator, in PowerShell, which is in effect an alias of the Out-File cmdlet:

    • Character encoding:

      • In Windows PowerShell (powershell.exe), it defaults to creating UTF16-LE ("Unicode") files, which most command-line utilities do not recognize.

      • In PowerShell (Core) 7+ (pwsh), it now - fortunately - defaults to BOM-less UTF-8, which is now the consistent default for all cmdlets, as well as for the PowerShell engine itself with respect to readings source code.

      • As an aside: In Windows PowerShell and PowerShell (Core) up to v7.3.x, the PowerShell pipeline and > never act as a raw byte conduit for relaying raw data from external programs: such data is invariably decoded into a .NET string first, which > / Out-File then write to a file based on their default encoding. See this answer for background.

    • Trailing newline:

      • In both editions, a platform-native newline is appended to the input (each input object) by default on writing the file, which can be suppressed with the -NoNewLine switch.

    The simplest solution is to use New-Item instead, as it - surprisingly - creates BOM-less UTF-8 files even in Windows PowerShell, and never appends a trailing newline (-ItemType File is implied):[1]

    $env:SSH_KEY | New-Item -Force ssh_key
    

    Note:

    • -Force is needed in order to ensure that a preexisting file by that name is overwritten. However, -Force also creates a non-existent target directory on demand, which > / Out-File won't do.

    • In Windows PowerShell, using Out-File (or Set-Content, which is generally preferable for text-only input)[2] with -Encoding Utf8 is not an option, because the resulting file is invariably created with a BOM.

    • Given that PowerShell (Core) 7+ defaults to BOM-less UTF-8, there you could alternatively use:

      $env:SSH_KEY | Set-Content -NoNewLine ssh_key
      
      • Assuming that the input string contains ASCII-range characters only, this will also work in Windows PowerShell, because Set-Content defaults to the ANSI code page, as determined by the legacy system locale, which is a superset of ASCII.[3]

    [1] Strictly speaking, this only applies if PowerShell's current location is one backed by the file-system provider - however, in practice it almost always is. If it isn't and you still want to create a file, either specify a full file-system path or prefix a relative one with FileSystem:: in order to target the file-system provider's current directory.

    [2] See this answer for background.

    [3] This implies, unfortunately, that in Windows PowerShell different cmdlets have different default encodings - see the bottom section of this answer for background.