running multiple bash commands with subprocess

Question

If I run echo a; echo b in bash the result will be that both commands are run. However if I use subprocess then the first command is run, printing out the whole of the rest of the line. The code below echos a; echo b instead of a b, how do I get it to run both commands?

import subprocess, shlex
def subprocess_cmd(command):
    process = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE)
    proc_stdout = process.communicate()[0].strip() 
    print proc_stdout
    
subprocess_cmd("echo a; echo b")

related: Here's how to run multiple shell commands (and optionally capture their output) concurrently — jfs
– jfs, Commented Jul 26, 2014 at 14:01

Nico Schlömer · Accepted Answer · 2021-05-18 19:57:43Z

137

You have to use shell=True in subprocess and no shlex.split:

import subprocess

command = "echo a; echo b"

ret = subprocess.run(command, capture_output=True, shell=True)

# before Python 3.7:
# ret = subprocess.run(command, stdout=subprocess.PIPE, shell=True)

print(ret.stdout.decode())

returns:

a
b

edited May 18, 2021 at 19:57

Nico Schlömer

59.6k35 gold badges216 silver badges291 bronze badges

answered Jul 19, 2013 at 10:11

bougui

3,8654 gold badges25 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Paul Over a year ago

Ah I see, I had shell=True tested but command.split() was breaking it (list). Also to note to other people reading, using shell=True is a security hazard, make sure you trust the input.

bougui Over a year ago

You cannot use command.split() with shell=True. Actually the argument of subprocess.Popen with shell=True must be a string and not a list.

Goblinhack Over a year ago

I found this to be a bit simpler and didn't have the garbling of the output: def do_shell(self, command): self.proc=subprocess.Popen(command,shell=True) self.proc.wait()

jfs Over a year ago

@Ravichandra: notice: bash in the title. On Windows, you probably want echo a & echo b command instead.

oldpride Over a year ago

@Milan, on windows, the default shell is batch, which does not honor ';' as a separator.

|

blong · Accepted Answer · 2024-04-30 02:07:36Z

35

I just stumbled on a situation where I needed to run a bunch of lines of bash code (not separated with semicolons) from within python. In this scenario the proposed solutions do not help. One approach would be to save a file and then run it with Popen, but it wasn't possible in my situation.

What I ended up doing is something like:

commands = '''
echo "a"
echo "b"
echo "c"
echo "d"
'''

process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE, text=True)
out, err = process.communicate(commands)
print(out)

So I first create the child bash process and after I tell it what to execute. This approach removes the limitations of passing the command directly to the Popen constructor.

The text=True addition is required for Python 3.

edited Apr 30, 2024 at 2:07

blong

2,7138 gold badges47 silver badges114 bronze badges

answered Mar 26, 2015 at 1:01

admenva

2,43120 silver badges12 bronze badges

4 Comments

jfs Over a year ago

subprocess.check_output(commands, shell=True) works just fine. If there are bash-isms in commands then pass executable='/bin/bash'.

Eyal Levin Over a year ago

For Python 3 you might want to use: out, err = process.communicate(commands.encode('utf-8')) and print(out.decode('utf-8'))

ssanch Over a year ago

If the commands are ; separated you might also want to add shell=True to the subprocess.Popen() line.

tripleee Over a year ago

Modern Python should use subprocess.run(commands, shell=True). Like the Popen documentation already tells you, avoid it when you can, such as when all you want to do is run a subprocess and wait for it to finish.

Benjamin Loison · Accepted Answer · 2025-04-26 15:39:33Z

23

Join commands with "&&".

os.system('echo a > outputa.txt && echo b > outputb.txt')

edited Apr 26 at 15:39

Benjamin Loison

5,7514 gold badges20 silver badges37 bronze badges

answered Aug 29, 2016 at 21:25

FrancisWolcott

3192 silver badges4 bronze badges

5 Comments

NateW Over a year ago

The second command would not run if the first command has an error. I would say use an ; instead.

Murphy Over a year ago

The important part here is to use os.system() instead of subprocess.

user1336619 Over a year ago

os.system is not a substitute for subprocess. os.system cannot do all the things that subprocess does.

Captain Jack Sparrow Over a year ago

Maybe not, but it can echo just fine.

Jonathan Rayner Over a year ago

subprocess is preferred to os.system, but no one has given an answer that works with subprocess and shell=False (or analogous).

Pierz · Accepted Answer · 2017-02-21 19:16:30Z

4

If you're only running the commands in one shot then you can just use subprocess.check_output convenience function:

def subprocess_cmd(command):
    output = subprocess.check_output(command, shell=True)
    print output

answered Feb 21, 2017 at 19:16

Pierz

8,40864 silver badges66 bronze badges

Comments

David.Zheng · Accepted Answer · 2013-07-19 10:00:50Z

1

>>> command = "echo a; echo b"
>>> shlex.split(command);
    ['echo', 'a; echo', 'b']

so, the problem is shlex module do not handle ";"

answered Jul 19, 2013 at 10:00

David.Zheng

9138 silver badges6 bronze badges

2 Comments

Paul Over a year ago

command.split() gives ['echo', 'a;', 'echo', 'b'] and also fails. `

David.Zheng Over a year ago

@bougui is right. Set "shell=True", the first argument of Popen need a command string like "echo a;echo b". and without "shell=True" the first argument of Popen should be a list, like:["echo","a"]

Benjamin Loison · Accepted Answer · 2025-04-26 15:38:59Z

1

Got errors like when I used capture_output=True

TypeError: __init__() got an unexpected keyword argument 'capture_output'

After made changes like as below and its works fine

import subprocess

command = '''ls'''

result = subprocess.run(command, stdout=subprocess.PIPE,shell=True)

print(result.stdout.splitlines())

edited Apr 26 at 15:38

Benjamin Loison

5,7514 gold badges20 silver badges37 bronze badges

answered Jun 17, 2022 at 4:11

Nanda Thota

4025 silver badges10 bronze badges

2 Comments

tripleee Over a year ago

capture_output was only introduced in Python 3.7, which is quite old already, but apparently yours is older still.

tripleee Over a year ago

(The shell=True is useless here; ls runs fine without a shell, though nominally you should then pass the command as a list ["ls"].)

Benjamin Loison · Accepted Answer · 2025-04-26 15:36:37Z

0

import subprocess
cmd = "vsish -e ls /vmkModules/lsom/disks/  | cut -d '/' -f 1  | while read diskID  ; do echo $diskID; vsish -e cat /vmkModules/lsom/disks/$diskID/virstoStats | grep -iE 'Delete pending |trims currently queued' ;  echo '====================' ;done ;"


def subprocess_cmd(command):
    process = subprocess.Popen(command,stdout=subprocess.PIPE, shell=True)
    proc_stdout = process.communicate()[0].strip()
    for line in proc_stdout.decode().split('\n'):
        print (line)

subprocess_cmd(cmd)

edited Apr 26 at 15:36

Benjamin Loison

5,7514 gold badges20 silver badges37 bronze badges

answered Jan 25, 2021 at 15:51

neelam gour

1

2 Comments

effprime Over a year ago

Welcome to StackOverflow! Please provide an explanation as to how the code you provided answers the original question.

neelam gour Over a year ago

Run commends with pipe(|)

Benjamin Loison · Accepted Answer · 2025-04-26 15:38:10Z

Answering this old question for the sake of completeness, as none of the answers suggest splitting the commands nor give an answer that works with shell=False

Practical solution: two calls to subprocess.run or subprocess.Popen + Popen.communicate
Note this ignores stderr

import subprocess, shlex

proc1 = subprocess.Popen(shlex.split("echo a"), stdout=subprocess.PIPE)
(proc1_stdout, _) = process.communicate()
print(proc1_stdout, end="")
proc2 = subprocess.Popen(shlex.split("echo b"), stdout=subprocess.PIPE)
(proc2_stdout, _) = process.communicate()
print(proc2_stdout, end="")

This handles executing chained commands through a single subprocess.run call decently well without relying on shell=True. However, this is needlessly complicated

import shlex
import subprocess
import sys
import warnings

def subprocess_cmd(command, text=True, posix=False):
    tokens: shlex.shlex = shlex.shlex(command, posix=posix, punctuation_chars=True)
    commands: list[list[str]] = [[]]
    separators: list[str] = [";"]

    for token in tokens:
        if token in ("(", ")"):
            warnings.warn("Parentheses are not implemented and will be ignored", RuntimeWarning)
        if token in ("|", ";", "||", "&&"):
            separators.append(token)
            commands.append([])
        else:
            commands[-1].append(token)

    out: str|bytes = "" if text else b""
    err: str|bytes = "" if text else b""
    ret: int = 0

    for (command, separator) in zip(commands, separators):
        if (separator in (";")) or (separator in ("|", "||") and not ret == 0) or (separator in ("&&") and ret == 0):
            if separator != "|": print(out, end="")
            print(err, end="", file=sys.stderr)

            if separator == "|":
                result = subprocess.run(command, input=out, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=text)
            else:
                result = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=text)

            out = result.stdout
            err = result.stderr
            ret = result.returncode

    print(out, end="")
    print(err, end="", file=sys.stderr)
    return ret

The boilerplate combo of Popen + communicate is unnecessary bloat. The standard library provides subprocess.run and the legacy subprocess.check_output and subprocess.check_command precisely so you don't need to write out this every time.

skyshifter · Accepted Answer · 2024-04-10 13:21:48Z

-2

What worked for me is very similar to what @Nico Schlömer and @bougui did.

But I had to add squared brackets to my commands and add comma in between each command.

import subprocess

command = ["echo a, echo b"]

after Python 3.7:

ret = subprocess.run(command, capture_output=True, shell=True)

before Python 3.7:

ret = subprocess.run(command, stdout=subprocess.PIPE, shell=True)


print(ret.stdout.splitlines())

edited Apr 10, 2024 at 13:21

user4136999

answered Apr 10, 2024 at 9:33

skyshifter

1

1 Comment

tripleee Over a year ago

This exhibits multiple misunderstandings of what the square brackets mean and what shell=True actually does. The comma is not a valid command separator in any standard shell so it's unclear under which circumstances this could ever have worked anywhere.

Collectives™ on Stack Overflow

running multiple bash commands with subprocess

9 Answers 9

7 Comments

4 Comments

5 Comments

Comments

2 Comments

2 Comments

2 Comments

1 Comment

after Python 3.7:

before Python 3.7:

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

9 Answers 9

7 Comments

4 Comments

5 Comments

Comments

2 Comments

2 Comments

2 Comments

1 Comment

after Python 3.7:

before Python 3.7:

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related