Subprocess and using the output file for another python script within Python script

Question

Using a single python script, I want the output of one program to be captured and consumed by a second program running in a separate process. I'm attempting to use subprocesses to run these two programs from within a single python script:

prog1 = 'python prog1.py'
args1 = 'input_1.txt'
proc1 = Popen(prog1 + ' ' + args1,stdout=PIPE,stderr=STDOUT, shell=True)
for i in proc1.stdout:
   print i
prog2 = 'python prog2.py'
args2 = i   # the input which is the output of prog1
proc2 = Popen(prog2 + ' ' + args2,stdout=PIPE,stderr=STDOUT, shell=True)
for m in proc2.stdout:
    print m

Do you mean prog1 input_1.txt | prog2 or do you mean prog2 <(prog1 input_1.txt)? — jfs
– jfs, Commented Jul 28, 2015 at 20:07

user590028 · Accepted Answer · 2015-07-28 19:26:10Z

2

Sounds like you want shell pipeline https://docs.python.org/2/library/subprocess.html#replacing-shell-pipeline

p1 = Popen(["python", "prog1.py"], stdout=PIPE)
p2 = Popen(["python", "prog2.py"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close()
output = p2.communicate()[0]

answered Jul 28, 2015 at 19:26

user590028

11.8k5 gold badges43 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

BioInformatician Over a year ago

thanks for your respond. prog2.py expects an input to be pass on the command line python prog2.py input2.txt and thus I got an error filename = sys.argv[1] IndexError: list index out of range from prog2.py

Community · Accepted Answer · 2017-05-23 10:26:49Z

If prog2 accepts input only as a filename i.e., if you need to emulate the bash process substitution: prog2 <(prog1 input_1.txt) then to avoid writing the output of prog1 subprocess to disk, you could use /dev/fd/N special filenames if your OS supports them:

#!/usr/bin/env python2
import sys
from subprocess import Popen, PIPE

prog1 = Popen([sys.executable, "prog1.py", "input_1.txt"], stdout=PIPE)
with prog1.stdout:
    cmd2 = [sys.executable, "prog2.py", "/dev/fd/%d" % prog1.stdout.fileno()]
    prog2 = Popen(cmd2, stdout=PIPE)
output = prog2.communicate()[0]
statuses = [prog1.wait(), prog2.returncode]

You should check whether prog2 supports a special filename '-' to read the input from stdin i.e., to emulate the pipeline: prog1 input_1.txt | prog2 -:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE

prog2 = Popen([sys.executable, "prog2.py", "-"], stdin=PIPE, stdout=PIPE)
prog1 = Popen([sys.executable, "prog1.py", "input_1.txt"], stdout=prog2.stdin)
output = prog2.communicate()[0]
statuses = [prog1.wait(), prog2.returncode]

Or (better) import the modules instead of running them as subprocesses and run the corresponding functions, use multiprocessing if necessary. See Call python script with input with in a python script using subprocess

Collectives™ on Stack Overflow

Subprocess and using the output file for another python script within Python script

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related