Using xargs to redirect input file parameter to a bash command

Question

I have a python command (that I cannot modify) that takes as input a file as and output the result:

  $ echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' > examples.jsonl
  $ python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz examples.jsonl

The usage is the following:

usage: python -m allennlp.run [command] predict [-h]
                                                [--output-file OUTPUT_FILE]
                                                [--weights-file WEIGHTS_FILE]
                                                [--batch-size BATCH_SIZE]
                                                [--silent]
                                                [--cuda-device CUDA_DEVICE]
                                                [-o OVERRIDES]
                                                [--include-package INCLUDE_PACKAGE]
                                                [--predictor PREDICTOR]
                                                archive_file input_file

Is there a way in bash to redirect the input to this command directly instead of echoing to a file? It seems that the command does not support pipe from stdin by the way, so the following will not work:

$ echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' | python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz

I have tried using xargs but I don't figure out the right way to handle the input_file parameter

mncl · Accepted Answer · 2018-04-16 09:20:45Z

3

Creates a temporary file descriptor that is deallocated when the command exits. It will work on most programs and replaces the argument of the file.

<(echo hello)

example

grep h <(echo hello)


python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz  <( echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' > examples.jsonl )

answered Apr 16, 2018 at 9:20

mncl

1611 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Ole Tange Over a year ago

When does that solution fail? It fails if allennlp seeks into the file. If called from bash, then this might work, too: echo ... | python -m allennlp... /dev/stdin (but this will also fail if allennlp seeks).

Ole Tange · Accepted Answer · 2018-04-23 00:44:24Z

1

If allennlp needs a file (i.e. cannot simply read from /dev/stdin), then this will work:

echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}' | 
  parallel --pipe -N1 --cat python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz {}

It is especially useful if you are going to run many allennlp with different input:

(echo '{"sentence": "Did Uriah honestly think he could beat The Legend of Zelda in under three hours?"}';
 echo '{"sentence": "its insatiable appetite is tempered by its fear of light."}';
 echo '{"sentence": "You were eaten by a grue"}';) |
  parallel --pipe -N1 --cat python -m allennlp.run predict https://s3-us-west-2.amazonaws.com/allennlp/models/ner-model-2018.02.12.tar.gz {}

answered Apr 23, 2018 at 0:44

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

1 Comment

loretoparisi Over a year ago

Thanks this is the most affordable solution. I will just add a reference to the GNU parallel jobs execution command: gnu.org/software/parallel

Collectives™ on Stack Overflow

Using xargs to redirect input file parameter to a bash command

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related