Running a Python Subprocess

Question

All,

I have read several threads on how to run subprocesses in python and none of them seem to help me. It's probably because I don't know how to use them properly. I have several methods that I would like to run at the same time rather than in sequence and I thought that the subprocess module would do this for me.

def services():
     services = [method1(),
            method2(),
            method3(),  
            mrthod4(),
            method5()]
     return services

def runAll():
    import subprocess
    for i in services():
        proc = subprocess.call(i,shell=True)

The problem with this approach is that method1() starts and method2() doesn't start until 1 finishes. I have tried several approaches including using subprocess.Popen[] in my services method with no luck. Can anyone lend me a hand on how to get methods 1-5 running at the same time?

Thanks, Adam

JMD · Accepted Answer · 2011-06-27 18:41:36Z

4

According to the Python documentation subprocess.call() waits for the command to complete. You should directly use the subprocess.Popen objects which will give you the flexibility you need.

answered Jun 27, 2011 at 18:41

JMD

2281 gold badge2 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

drone115b · Accepted Answer · 2011-07-28 17:31:48Z

1

In python 3.2.x the concurrent futures module makes this sort of things very easy.

answered Jul 28, 2011 at 17:31

drone115b

2081 silver badge6 bronze badges

Comments

rafalotufo · Accepted Answer · 2011-06-27 18:42:44Z

1

Python threads are more appropriate to what you are looking for: http://docs.python.org/library/threading.html or even the multiprocessing module: http://docs.python.org/library/multiprocessing.html#module-multiprocessing.

answered Jun 27, 2011 at 18:42

rafalotufo

3,9524 gold badges27 silver badges28 bronze badges

1 Comment

Fred Foo Over a year ago

Threads are likely not appropriate at all, unless the tasks are I/O-bound or the OP is running Jython. Multiprocessing is a better idea.

Fred Foo · Accepted Answer · 2011-06-27 18:45:24Z

1

By saying method1(), you're calling the function and waiting for it to return. (It's a function, not a method.)

If you just want to run a bunch of heavy-duty function in parallel and collect their result, you can use joblib:

from joblib import Parallel, delayed

functions = [fn1, fn2, fn3, fn4]

results = Parallel(n_jobs=4)(delayed(f)() for f in functions)

answered Jun 27, 2011 at 18:45

Fred Foo

365k80 gold badges765 silver badges852 bronze badges

Comments

Corey Goldberg · Accepted Answer · 2011-06-27 19:13:18Z

1

subprocess.call() blocks until the process completes.

multiprocessing sounds more appropriate for what you are doing.

for example:

from multiprocessing import Process

def f1():
    while True:
        print 'foo'

def f2():
    while True:
        print 'bar'

def f3():
    while True:
        print 'baz'

if __name__ == '__main__':
    for func in (f1, f2, f3):
        Process(target=func).start()

edited Jun 27, 2011 at 19:13

answered Jun 27, 2011 at 18:56

Corey Goldberg

61.4k30 gold badges135 silver badges147 bronze badges

Comments

Mikola · Accepted Answer · 2011-06-27 20:26:33Z

1

You need to use & to execute them asynchronously. Here is an example:

 subprocess.call("./foo1&", shell=True)
 subprocess.call("./foo2&", shell=True)

This is just like the ordinary unix shell.

EDIT: Though there are multiple, much better ways to do this. See the other answers for some examples.

edited Jun 27, 2011 at 20:26

answered Jun 27, 2011 at 18:42

Mikola

9,3542 gold badges36 silver badges42 bronze badges

Comments

Senthil Kumaran · Accepted Answer · 2011-06-27 18:42:56Z

0

subprocess does not make the processes asynchronous. What you are trying to achieve can be done using multithreading or multiprocessing module.

answered Jun 27, 2011 at 18:42

Senthil Kumaran

57.3k15 gold badges99 silver badges139 bronze badges

Comments

j13r · Accepted Answer · 2012-03-18 18:51:27Z

I had a similar problem recently, and solved it like this:

from multiprocessing import Pool
def parallelfuncs(funcs, args, results, bad_value = None):
    p = Pool()
    waiters = []
    for f in funcs:
        waiters.append(p.apply_async(f, args, callback = results.append))
    p.close()
    for w in waiters:
        if w.get()[0] == bad_value:
            p.terminate()
    return p

The nice thing is that the functions funcs are executed in parallel on args (kind of the reverse of map), and the result returned. The Pool of multiprocessing uses all processors and handles job execution.

w.get blocks, if that wasn't clear.

For your use case, you would call

results = []
parallelfuncs(services, args, results).join()
print results

Collectives™ on Stack Overflow

Running a Python Subprocess

8 Answers 8

Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

8 Answers 8

Comments

Comments

1 Comment

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related