Search code examples
bashpipexargs

Using xargs to redirect input file parameter to a bash command

I have a python command (that I cannot modify) that takes as input a file as and output the result:

  $ echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' > examples.jsonl
  $ python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz examples.jsonl

The usage is the following:

usage: python -m allennlp.run [command] predict [-h]
                                                [--output-file OUTPUT_FILE]
                                                [--weights-file WEIGHTS_FILE]
                                                [--batch-size BATCH_SIZE]
                                                [--silent]
                                                [--cuda-device CUDA_DEVICE]
                                                [-o OVERRIDES]
                                                [--include-package INCLUDE_PACKAGE]
                                                [--predictor PREDICTOR]
                                                archive_file input_file

Is there a way in bash to redirect the input to this command directly instead of echoing to a file? It seems that the command does not support pipe from stdin by the way, so the following will not work:

$ echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' | python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz

I have tried using xargs but I don't figure out the right way to handle the input_file parameter

Solution

  • If allennlp needs a file (i.e. cannot simply read from /dev/stdin), then this will work:

    echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' | 
  parallel --pipe -N1 --cat python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz {}

    It is especially useful if you are going to run many allennlp with different input:

    (echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}';
 echo '{"sentence": "its insatiable appetite is tempered by its fear of light."}';
 echo '{"sentence": "You were eaten by a grue"}';) |
  parallel --pipe -N1 --cat python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz {}