Write background command output (stdout) to file during execution

Question

I have a command that takes a long time that I like to run in the background like this:

python3 script.py -f input.key -o output >> logs/script.log 2>&1 &

This works perfectly in the sense that the command is indeed in the background and I can check the output and potential errors later.

The main problem is the output is only appended after the command is completely finished, whereas I would like to have up-to-date log messages so I check the progress.

So currently the log would be empty and than suddenly at 08:30 two lines would appear:

[08:00] Script starting...
[08:30] Script finished!

Instead, I would like to have output saved to file before the command is completely finished.

See this and this post. I guess in mac pipe doesn’t do buffering because I am getting realtime output. So can’t test it. — Mihir Luthra
– Mihir Luthra, Commented Jul 20, 2019 at 10:07
Can you show us how you print from python? If I remember some calls are buffered and you might have to call sys.stdout.flush() — urban
– urban, Commented Jul 20, 2019 at 10:50
I use regular print() statements. Buffering was indeed the problem, and your solution might work too. — Roberto
– Roberto, Commented Jul 20, 2019 at 14:12

l'L'l · Accepted Answer · 2019-07-20 10:53:17Z

3

Since you are calling a Python script you would want to use the -u option, which forces the stdout and stderr streams to be unbuffered.

$ python3 -u script.py -f input.key -o output >> logs/script.log 2>&1 &

You can check the log periodically using cat or realtime in combination with watch:

$ watch cat logs/script.log

↳ https://docs.python.org/3.7/using/cmdline.html#cmdoption-u

answered Jul 20, 2019 at 10:53

l'L'l

47.5k12 gold badges102 silver badges154 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Roberto Over a year ago

Simple effective answer, thank you. Works both on Python 3.5 and 3.7 on my systems.

Collectives™ on Stack Overflow

Write background command output (stdout) to file during execution

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related