Run Python script within Python by using `subprocess.Popen` in real time

Question

I want to run a Python script (or any executable, for that manner) from a python script and get the output in real time. I have followed many tutorials, and my current code looks like this:

import subprocess
with open("test2", "w") as f:
    f.write("""import time
print('start')
time.sleep(5)
print('done')""")

process = subprocess.Popen(['python3', "test2"], stdout=subprocess.PIPE)
while True:
        output = process.stdout.readline()
        if output == '' and process.poll() is not None:
            break
        if output:
            print(output.strip())
        rc = process.poll()

The first bit just creates the file that will be run, for clarity's sake.

I have two problems with this code:

It does not give the output in real time. It waits untill the process has finished.
It does not terminate the loop once the process has finished.

Any help would be very welcome.

EDIT: Thanks to @JohnAnderson for the fix to the first problem: replacing if output == '' and process.poll() is not None: with if output == b'' and process.poll() is not None:

Please see my latest edit if the code does not run correctly. — Jachdich
– Jachdich, Commented Feb 26, 2019 at 19:02
I presume you want to use output for something else? Because otherwise just leaving stdout set to None should work for you (still line buffered by default, may very depending on platform). Otherwise you need stdout attached to a thread that consumes console output storing and printing content as it passes it through. — Ondrej K.
– Ondrej K., Commented Feb 26, 2019 at 19:19
@OndrejK. Yes, I want to use output later on in the program. — Jachdich
– Jachdich, Commented Feb 26, 2019 at 19:21
On Ubuntu, I get output in real time. Are you on Windows? Also, you need to change your if to if output == b'' and process.poll() is not None: otherwise it will never be True. — John Anderson
– John Anderson, Commented Feb 26, 2019 at 19:30
The problem is not your input, but your python subprocess output which is buffered, add -u to the python call created with subprocess to turn buffering off (['python3', '-u', 'test2']) and you should see lines appear as they are "printed". Default behavior is: stdout to console -> line buffered, stdout to anything else -> default buffer (I reckon in your case 4KB). — Ondrej K.
– Ondrej K., Commented Feb 26, 2019 at 21:23

Ondrej K. · Accepted Answer · 2019-02-27 07:45:28Z

Last night I've set out to do this using a pipe:

import os
import subprocess

with open("test2", "w") as f:
    f.write("""import time
print('start')
time.sleep(2)
print('done')""")

(readend, writeend) = os.pipe()

p = subprocess.Popen(['python3', '-u', 'test2'], stdout=writeend, bufsize=0)
still_open = True
output = ""
output_buf = os.read(readend, 1).decode()
while output_buf:
    print(output_buf, end="")
    output += output_buf
    if still_open and p.poll() is not None:
        os.close(writeend)
        still_open = False
    output_buf = os.read(readend, 1).decode()

Forcing buffering out of the picture and reading one character at the time (to make sure we do not block writes from the process having filled a buffer), closing the writing end when process finishes to make sure read catches the EOF correctly. Having looked at the subprocess though that turned out to be a bit of an overkill. With PIPE you get most of that for free and I ended with this which seems to work fine (call read as many times as necessary to keep emptying the pipe) with just this and assuming the process finished, you do not have to worry about polling it and/or making sure the write end of the pipe is closed to correctly detect EOF and get out of the loop:

p = subprocess.Popen(['python3', '-u', 'test2'],
                     stdout=subprocess.PIPE, bufsize=1,
                     universal_newlines=True)
output = ""
output_buf = p.stdout.readline()
while output_buf:
    print(output_buf, end="")
    output += output_buf
    output_buf = p.stdout.readline()

This is a bit less "real-time" as it is basically line buffered.

Note: I've added -u to you Python call, as you need to also make sure your called process' buffering does not get in the way.

Collectives™ on Stack Overflow

Run Python script within Python by using `subprocess.Popen` in real time

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related