Capturing LIVE output of shell script while running it in python

Question

I am writing a python script to ssh into a linux server and execute a shell script that is already stored on the linux server.

Here is what my code look like so far

command = ['ssh into the remote server',
           'cd into the directory of the shell script,
           './running the shell script',
           ]

process = subprocess.Popen(command,
                            shell=True,
                            stdin=subprocess.PIPE,
                            stdout=subprocess.PIPE,
                            stderr=subprocess.PIPE)

err, out = process.communicate()

if out: 
    print "standard output of subprocess is : "
    print out
if err:
    print "standard error of subprocess is : "
    print err
print "returncode of subprocess: "
print process.returncode

1st question: I can obtain the output of my shell scripts through stderr, but I only obtain it after the entire shell script has finished executing. So if the shell script takes 10 minutes to finish, I only get to see the output of the shell script after 10 minutes. I want to have the output of my shell scripts return line by line to me just as if I was executing the script manually in the remote server. Can this be done?

2nd question: as you can see, I have three commands in my command list (which is only a small portion of all my commands,) if I put all my commands in the list, I only obtain the output of ALL my commands through stdout ONLY when all my commands has finished executing. If my 1st question cannot be done, is there a way to at least obtain the output of each command after each one has been executed instead of receiving them all at once only when all the commands has finished being executed.

can you elaborate on that? this script has to be very robust, so I am trying to use all built in modules of python, because if someone else want to run this script, that person would have to install paramiko before running this script. So if there is a way to do it without importing modules, I'd prefer doing it that way even if it might be harder. — Shuixi Li
– Shuixi Li, Commented Jan 29, 2015 at 12:18
I see you have problems reading from subprocess pipe, but paramiko allows you to read directly from SSH channel (from network). In my opinion lesser wrappers is better. For 'oneliners' or similiar scripts of course, it would be overengineering and subprocess is enough. — myaut
– myaut, Commented Jan 29, 2015 at 12:47

Community · Accepted Answer · 2017-05-23 12:21:42Z

To see the output immediately, don't redirect it:

from subprocess import Popen, PIPE

p = Popen(['ssh', 'user@hostname'], stdin=PIPE) 
p.communicate(b"""cd ..
echo 1st command
echo 2nd command
echo ...
""")

If you want both to capture the "live" output in a variable and to display it in the terminal then the solution depends on whether you need to handle stdin/stdout/stderr concurrently.

If input is small and you want to combine stdout/stderr then you could pass all commands at once and read the merged output line-by-line:

from subprocess import Popen, PIPE, STDOUT

p = Popen(['ssh', 'user@hostname'], stdin=PIPE,
          stdout=PIPE, stderr=STDOUT, bufsize=1) 
p.stdin.write(b"""cd ..
echo 1st command
echo 2nd command
echo ...
""")
p.stdin.close() # no more input

lines = [] # store output here
for line in iter(p.stdout.readline, b''): # newline=b'\n'
    lines.append(line) # capture for later
    print line, # display now
p.stdout.close()
p.wait()

If you want to capture "live" stdout/stderr separately, see:

Siguza · Accepted Answer · 2015-01-29 12:19:02Z

0

I'm not entirely sure, but maybe you get instant output if you pass the other two commands as arguments to ssh:

command = 'ssh [email protected] \'cd some/path/on/your/server; ./run-the-script.sh\''

The way I understand it, Python first reads and processes all the input and only then returns output. I'm not too familiar with Python, so I might be wrong on this, but if I'm right, this should help.

answered Jan 29, 2015 at 12:19

Siguza

24.7k6 gold badges58 silver badges105 bronze badges

Comments

Dima Tisnek · Accepted Answer · 2015-01-29 12:36:03Z

-1

Don't call .communicate() -- that waits for the process to finish.

Instead, keep reading data from .stdout pipe.

Simple example:

In [1]: import subprocess
In [2]: p = subprocess.Popen(["find", "/"], stdout=subprocess.PIPE)


In [3]: p.stdout
Out[3]: <open file '<fdopen>', mode 'rb' at 0x7f590446dc00>

In [4]: p.stdout.readline()
Out[4]: '/\n'

In [5]: p.stdout.readline()
Out[5]: '/var\n'

In [6]: p.stdout.readline()
Out[6]: '/var/games\n'

edited Jan 29, 2015 at 12:36

answered Jan 29, 2015 at 12:22

Dima Tisnek

11.9k4 gold badges73 silver badges129 bronze badges

1 Comment

Shuixi Li Over a year ago

can you show me that in code? I'm not sure how to approach that

Collectives™ on Stack Overflow

Capturing LIVE output of shell script while running it in python

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related