How to create lists by running functions in parallel in python

Question

I want to create two lists by running two functions (returning a value each for every run) in parallel. My code below works, but is still taking too much time. Is there a more efficient way to parallelize this code?

import time
from joblib import Parallel, delayed
 
catchments = 50   #define number of catchments to plot here
randomlist = random.sample(range(2, 2100), catchments)

def budx(i):   #Time-taking task....
    try:        
        catch = slicecatch(i)
        return catch.PETNatVeg.mean().values/catch.Prec.mean().values
    except IndexError as e:
        pass    
        
def budy(i):   #Time-taking task....
    try:        
        catch = slicecatch(i)
        return catch.TotalET.mean().values/catch.Prec.mean().values
    except IndexError as e:
        pass 
    
        
 
start_time = time.perf_counter()

bud_x = Parallel(n_jobs=-1)(delayed(budx)(i) for i in randomlist)
bud_y = Parallel(n_jobs=-1)(delayed(budy)(i) for i in randomlist) 

finish_time = time.perf_counter()

larsks · Accepted Answer · 2022-11-08 23:34:01Z

1

The way you've written your code, you're first running all your budx instances, waiting for them to complete, and only then running your budy instances. That is, you are sequentially running two sets of parallel tasks.

Here's one possible way of doing that, noting that (a) I was not previously familiar with joblib, so there may be a more canonical form, and (b) I've replaced your budx and budy implementations with code that I can actually run:

import time
from joblib import Parallel, delayed
import random
 
catchments = 50   #define number of catchments to plot here
randomlist = random.sample(range(2, 2100), catchments)

def budx(i):   #Time-taking task....
    print("start budx", i)
    time.sleep(random.randint(0, 10))
    print("end budx", i)
    return ("budx", i)

def budy(i):   #Time-taking task....
    print("start budy", i)
    time.sleep(random.randint(0, 10))
    print("end budy", i)
    return ("budy", i)
 
start_time = time.perf_counter()

results = Parallel(n_jobs=-1)(
        [delayed(budx)(i) for i in range(5)] +
        [delayed(budy)(i) for i in range(5)])

finish_time = time.perf_counter()

print("total time:", finish_time - start_time)
print("results", results)

If I were writing this I would probably opt for native Python tools like concurrent.futures rather than a third-party module like joblib (unless there are additional features provided by joblib that make your life easier).

answered Nov 8, 2022 at 23:34

larsks

318k49 gold badges473 silver badges482 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Zeeshan Asghar Over a year ago

Thank you larsks! Your improvement certainly helps and the code is several times faster now :)

Collectives™ on Stack Overflow

How to create lists by running functions in parallel in python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related