Search code examples
bashshellpipewget

Pipe wget into sh - issue with line breaks

I'm trying to download a bash script using wget and pipe it over to sh but I'm running into problems. It works fine in separate steps (wget to download, then execute), but not when piping to sh. The script can be found in this gist, but I'll include it at the bottom of this question for convenience.

I'm running the following to:

  • download the script
  • pipe it to sh
  • pass in a parameter of linux
wget -q --no-cache https://gist.githubusercontent.com/devklick/be9cbe0b384958a7e7ce115f7df25e5a/raw/27153ef3109e7561e45f0fe43e9a10276da78252/install.sh -O - | sh -s linux

When I run the above, the script fails trying to parse some JSON returned from Githubs REST API. We can see that the last few lines logged look like this:

GDMan/zipball/v1.3.1", "body": "## [1.3.1](https://github.com/devklick/GDMan/compare/v1.3.0...v1.3.1) (2024-05-07)


### Bug Fixes

* fixes [#15](https://github.com/devklick/GDMan/issues/15), --help is not a known command ([ddd78f2](https://github.com/devklick/GDMan/commit/ddd78f2a92da239bab9a08658b732d1460b19dc3))



" }
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 150, column 1
Found download ''

I'm guessing that the line breaks shown here are the problem. However this problem does not exist when using wget to download and save the file, then execute it:

wget -q --no-cache https://gist.githubusercontent.com/devklick/be9cbe0b384958a7e7ce115f7df25e5a/raw/27153ef3109e7561e45f0fe43e9a10276da78252/install.sh
chmod +x ./install.sh
./install.sh linux

Running these steps separately we can see that the last few lines output looks a little different:

GDMan/zipball/v1.3.1", "body": "## [1.3.1](https://github.com/devklick/GDMan/compare/v1.3.0...v1.3.1) (2024-05-07)\n\n\n### Bug Fixes\n\n* fixes [#15](https://github.com/devklick/GDMan/issues/15), --help is not a known command ([ddd78f2](https://github.com/devklick/GDMan/commit/ddd78f2a92da239bab9a08658b732d1460b19dc3))\n\n\n\n" }
Found download 'https://github.com/devklick/GDMan/releases/download/v1.3.1/GDMan_v1.3.1_linux-x64.zip'

So the question is; why is my script behaving differently when written to the file system vs in memory, and how can I fix this to solve the JSON line break issue without first saving the script before running it?

I know I can get around this by running the steps individually but chaining them with &&, but I'm invested in this issue now and would really like to know what the problem/solution is.

Bash script in question:

#!/bin/bash

target_os=$1
echo "Target OS '$target_os'"

zip_path="$HOME/Downloads/gdman.zip"
echo "Asset download path '$zip_path'"

install_dir="$HOME/gdman"
echo "App install path '$zip_path'"

# Download the latest release information from GitHub API
echo "Finding latest version"
response=$(curl -s https://api.github.com/repos/devklick/GDMan/releases/latest)
echo $response

# Extract the browser download URL for the asset
download_url=$(echo "$response" | jq -r '.assets[] | select(.name | test("'${target_os}'")) | .browser_download_url')
echo "Found download '$download_url'";
Solution

  • This happens because you use sh (likely dash) in one case and bash in the other, and echo is not portable between shells.

    Your file contains \n\n in a JSON string, which gives different results in each:

    $ bash -c 'echo "\n\n"'
\n\n
$ dash -c 'echo "\n\n"'
(blank line)
(blank line)

    Here's POSIX:

    It is not possible to use echo portably across all POSIX systems unless both -n (as the first argument) and escape sequences are omitted.

    If you would like your script to work across multiple shells, you can replace echo with printf or simply pipe directly from curl to jq.