0

I have a shell script called load_data.sh that runs inside a shell script called shell.sh

The contents of shell.sh

xargs --max-procs 10 -n 1 sh load_data.sh < tables.txt

This shell script runs on 10 tables at the same time in the tables.txt

Now I want to collect the Full logs of the load_data.sh So I did

xargs --max-procs 10 -n 1 sh load_data.sh < tables.txt |& tee-a logs.txt

But I am getting a mix of all the logs. What I want is the logs should be 1st table log then 2nd table log and then 3rd table logs and so on...

Is it possible to achieve that. If so how can I achieve that?

Improve this question
4
  • Cant you create log files with load_data.sh ? This will solve your problem I think. Commented Apr 3, 2017 at 0:20
  • Because you are running the processes in parallel the only way to separate the output is to create a separate log for each process and then concatenate them at completion. Commented Apr 3, 2017 at 0:21
  • @JohnC I am a bit new to shell scripting Could you please let me know how to do this Commented Apr 3, 2017 at 2:40
  • @suleiman I put the same logic in load_data.sh file but it gave all the logs in zigzag order Commented Apr 3, 2017 at 2:41

1 Answer 1

Reset to default
1

You can solve your problem by creating a separate logfile for each time your script is run. To get the logfiles to be created in sequence you can use the 'nl' utility to number each line of input.

nl -n rz tables.txt | xargs -n 2 --max-procs 10 sh -c './load_data.sh "$1" > log-$0'

Will produced logfiles in sequence 

log-001
log-002
log-003 
..

To turn that back into one file you can just use 'cat'

cat log* > result
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

How can we add a date directory at the end > log-$0' so that all the log files go to the directory of that particular date

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.