Executing bash script on multiple lines inside multiple files in parallel using GNU parallel

Question

I want to use GNU parallel for the following problem:

I have a few files each with several lines of text. I would like to understand how I can run a script (code.sh) on each line of text of each file and for each file in parallel. I should be able to write out the output of the operation on each input file to an output file with a different extension.

Seems this is a case of multiple parallel commands running parallel over all files and then running parallel for all lines inside each file.

This is what I used:

ls mydata_* |
    parallel -j+0 'cat {} | parallel -I ./explore-bash.sh > {.}.out'

I do not know how to do this using GNU parallel. Please help.

Have you walked through the tutorial? man parallel_tutorial or gnu.org/software/parallel/parallel_tutorial.html — Ole Tange
– Ole Tange, Commented Jun 1, 2017 at 13:17

Ole Tange · Accepted Answer · 2017-06-01 13:25:53Z

3

Your solution seems reasonable. You just need to remove -I:

ls mydata_* | parallel -j+0 'cat {} | parallel ./explore-bash.sh > {.}.out'

Depending on your setup this may be faster as it will only run n jobs, where as the solution above will run n*n jobs in parallel (n = number of cores):

ls mydata_* | parallel -j1 'cat {} | parallel ./explore-bash.sh > {.}.out'

answered Jun 1, 2017 at 13:25

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Executing bash script on multiple lines inside multiple files in parallel using GNU parallel

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related