Search code examples
iojq

What is the difference between `jq -nR inputs` and `jq -R split("\n")`?


At the first glace using inputs and splits seem to produce the same result:

$ seq 3 | jq -nR 'inputs|tonumber'
1
2
3
$ seq 3 | jq -R 'splits("\n")|tonumber'
1
2
3

But when I try to reduce the input, inputs works:

$ seq 3 | jq -nR 'reduce inputs as $_ (0; . + ($_|tonumber))'
6

But splits does not work:

$ seq 3 | jq -R 'reduce splits("\n") as $_ (0; . + ($_|tonumber))'
1
2
3

Why is splits not working?


Solution

  • The filter following seq 3 | jq -R … actually receives not one but three inputs because the -R flag makes jq provide one string for each line of input. In fact, the manual says:

    --raw-input / -R:
    
        Don´t parse the input as JSON. Instead, each line of text is passed
        to the filter as a string. […]
    

    But then, when applied to each line individually, splits("\n") has no effect anymore (with final newlines stripped from each input, only one partition, namely the full input string, is produced), and seq 3 | jq -R 'splits("\n")|tonumber' becomes no different from just seq 3 | jq -R 'tonumber', which just coincidentally matches your expected outcome.

    However, as the manual continues with the definition of the -R flag:

    --raw-input / -R:
    
        […] If combined with `--slurp`, then the entire input is passed to
        the filter as a single long string.
    

    That's what you intended to compare the inputs variant against, I suppose. And running seq 3 | jq -Rs 'splits("\n")|tonumber' instead indeed receives only one input string (namely "1\n2\n3\n"), which is split at newlines into a stream of substrings, then each of them is converted into a number. This produces the expected output -- and also throws an error! That's because the split actually produced four substrings, as the input contained three newline characters, with the fourth output being the empty string, which tonumber cannot handle. To catch up, add the error suppression operator ?, and seq 3 | jq -Rs 'splits("\n")|tonumber?' will work as intended, producing the expecetd output without errors.

    As for computing the sum of these values using reduce, this explains why seq 3 | jq -R 'reduce …' actually runs the reduction three times, independently from each other and with one iteration each, resulting in the simple reproduction of the individual corresponding inputs (each number is separately added to the initial 0 value). Using -Rs, on the other hand, does the job, but the "fourth" value still needs more treatment. Only suppressing the conversion error would produce empty in that case, spoiling the numeric sum, and rendering the overall output null. Instead, switch to the neutral summand 0 to retain the sum, provided by the alternative operator //, and now you can also reduce the output of seq 3 into the expected output of 6 using the -R (and -s) flags:

    seq 3 | jq -Rs 'reduce splits("\n") as $_ (0; . + ($_|tonumber? // 0))'
    
    6
    

    Demo