Python script differs output between terminal and redirection to a file

Question

I am writing a large python script with various subprocess.call to execute commands available in the system, and I have an issue because the output differs if it is printed into the terminal or if is redirected to a file.

For reproduce the problem, this is a small part of the script:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from subprocess import call

print "Hello at first"
call(["rsync", "-aHvz", "root@server1:/tmp/a", '.'])
print "Hello at the end"

Executing it from terminal return that in the correct order, print + rsync + print:

$ python keepconf.py
Hello at the first
receiving incremental file list

sent 12 bytes  received 123 bytes  270.00 bytes/sec
total size is 55858143  speedup is 413764.02
Hello at the end

Executing the same, but redirecting the output to a file:

$ python keepconf.py  > /tmp/output
$ cat /tmp/output
receiving incremental file list

sent 12 bytes  received 123 bytes  270.00 bytes/sec
total size is 55858143  speedup is 413764.02
Hello at the first
Hello at the end

The order now is rsync + print + print. Why is this behaviour?

And with sys.stdout.flush() after your prints ?

Gerard Rozsavolgyi
– Gerard Rozsavolgyi

2015-01-15 16:03:07 +00:00
Commented Jan 15, 2015 at 16:03 — Gerard Rozsavolgyi
– Gerard Rozsavolgyi, Commented Jan 15, 2015 at 16:03

Martijn Pieters · Accepted Answer · 2015-01-15 18:22:23Z

4

Output for a terminal (or more precisely, a tty) is typically opened in line buffer mode in Python. When you use a pipe, Python'll use a different buffer, of a fixed size.

This means that when you write text with a newline, the buffer is automatically flushed when printing to a terminal, but to a pipe it'll only be flushed when the buffer is full or a flush is forced (such as when the Python script exits).

In other words, when writing to the terminal the first print line is flushed out to the terminal before the rsync command is run. When redirecting to a pipe, the text is kept in a buffer, the rsync command output is run (writing to the pipe when it flushes, at least once at the end but possibly more often), after which you write some more to the buffer and that buffer is flushed to the pipe when Python exists.

You can force the flush manually:

import sys

# ....
print "Hello at first"
sys.stdout.flush()
call(["rsync", "-aHvz", "root@server1:/tmp/a", '.'])
print "Hello at the end"
sys.stdout.flush()

edited Jan 15, 2015 at 18:22

answered Jan 15, 2015 at 16:03

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jfs Over a year ago

upvote. Correction: sys.stdout (and C stdout stream where the subprocess writes) may be line-buffered if it refers to a tty (common for stdio-based programs such as python2). Terminal editing mode (raw, line) is unrelated here.

Martijn Pieters Over a year ago

@J.F.Sebastian: Qualified it; I habitually refer to it a a terminal for short, but having actually used physical terminals, you are correct that that is not precise enough.

jfs Over a year ago

no. I meant that sys.stdout Python object, C stdout (FILE*) (program properties -- inside) are different from terminal (stty may change its parameters). The latter exists before/during/after the program (outside). The buffers are inside the processes.

Martijn Pieters Over a year ago

@J.F.Sebastian: right, I am talking about the buffers in the Python process here.

Collectives™ on Stack Overflow

Python script differs output between terminal and redirection to a file

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related