Search code examples
curljqxargsmsys2

Downloading file with curl fails when called through xargs

I am using curl to download a file from Artifactory. I am on windows 10 using MSYS2.

curl --version
curl 7.68.0 (x86_64-pc-msys) libcurl/7.68.0 OpenSSL/1.1.1d zlib/1.2.11 brotli/1.0.7 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.2.0) libssh2/1.9.0 nghttp2/1.40.0
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS brotli Debug GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz Metalink NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP TrackMemory UnixSockets

xargs --version
xargs (GNU findutils) 4.7.0
...

which xargs
xargs is an external : C:\msys64\usr\bin\xargs.exe

This is my command:

curl "http://artifactory.xxx.xxx.com:8080/artifactory/api/storage/generic-tools/xxx-xxx/?lastModified" | jq .uri | xargs curl

Curl downloads the json file and jq extract the url for the json file containing with the library information. However the second curl call fails with:

curl: (3) URL using bad/illegal format or missing URL

Changing the xargs command to xargs echo curl gives the expected output. When running this debug output on the command line the correct file is downloaded without any errors.

I can also create a config file that can download the json file:

echo.|set /P =URL =  > latest.txt
curl ... | jq .uri >> latest.txt
curl --config latest.txt

Why is the piped xargs command not working but the command line command as well as the config file option is working correctly?

Solution

  • The Windows binary from the JQ homepage follows the DOS End Of Line conventions and uses CRLF. the xargs binary from MSys2 might expect Unix convention for EOLs which is just LF. So curl sees an additional stray CR character which it assumes to be part of the URL.

    Check that theory by using something like this and check for stray CR (0x0d) characters between two URLs:

    jq '.uri' input.json  |xargs  echo | hexdump -C

    00000000  68 74 74 70 3a 2f 2f 65  78 61 6d 70 6c 65 2e 63  |http://example.c|
00000010  6f 6d 2f*0d*20 68 74 74  70 3a 2f 2f 65 78 61 6d  |om/. http://exam|
00000020  70 6c 65 2e 63 6f 6d 2f  0d 0a                    |ple.com/..|
0000002a