Parallelize a function which appends lists in python

Question

PROBLEM: I have a for loop which takes a very long time to run. I was hoping to parallelize it to speed it up. However I feel that I have been complicating the problem more than I should be. The loop will be given below. Each variable (rtran-MDvals) is constant. For a bit extra information Nr is equal to 100. The two functions M_B and M_D are fairly complicated but essentially only the rv term is different since it is related to j by the list rvm. Ill show my attempt to parallelize it below the code.

CODE:

MBvals = []
MDvals = []
radialvalue  = 0
rv           = 0
rtran        = 0
j            = 0 


def tran_for_loop(j,rtran,Nr,Rb,n,b_n,M_B_tot,MBvals,Rd,M_D_tot,MDvals):
    for j in range(Nr):
        
        rv=rvm[j]                                 # the jth term from the list rvm is assigned to rv
        MB_rv = M_B(rv,Rb,n,b_n, M_B_tot)         # rv is inputed into the function M_B 
        MBvals.append(MB_rv)                      # the value of that function MB_rv is appended to the list MBvals
        MD_rv = M_D(rv,Rd,M_D_tot)                # rv is inputed into the function M_D
        MDvals.append(MD_rv)                      # the value of that function MD_rv is appended to the list MBvals
                                                  # outputs are the two lists MBvals and MDvals

ATTEMPT:

if __name__ =='__main__':
    pool = mp.Pool(mp.cpu_count())
    pool.map(tran_for_loop, [j for j in range(Nr)])
    pool.close()

ERROR

INDEXERROR: LIST INDEX IS OUT OF RANGE

I tried debugging to figure out if this error is coming after the Parallelization or during, and as far as I can tell it is during. I have a feeling I either just completely messed up the syntax of I'm not able to append lists in Parallel like I think I can. Any help would be appreciated. If something is not clear please let me know.

EDITS

rvm is a list of 100 values ranging from .1 to 100.

I think you are missing where rvm is defined. That seems to be an important variable in this problem. Can you edit the question to include the definition of rvm? — David Kaftan
– David Kaftan, Commented Jun 8, 2021 at 21:05
don't append, create the list ahead of time and populate the values based on the index — Cireo
– Cireo, Commented Jun 8, 2021 at 21:05

David Kaftan · Accepted Answer · 2021-06-08 21:36:13Z

2

I think you are close, but you may be misunderstanding how pool.map works. pool.map takes two arguments. First, a Callable (ie, a function), and second, a list. pool.map will call you function using each element of the array, and return an output-list corresponding to the outputs of said function for each element of the input-list. So there is no need to have a for-loop in your function. See the changes I made to your code:

import multiprocessing as mp

# dummy functions for demonstration
def M_B(rv, *args):
    return rv

def M_D(rv, *args):
    return 10*rv

# These are made up numbers
# I don't know what you want them to be,
# But if they are constant for each run,
# Define them before run_experiment
Rb = 10 
n = 11
b_n = 12
M_B_tot = 13
Rd = 14
M_D_tot = 15

def run_experiment(rv):
    MB_rv = M_B(rv, Rb, n, b_n, M_B_tot)
    MD_rv = M_D(rv,Rd,M_D_tot)
    return MB_rv, MD_rv

rvm = [1.,2.,3.,4.,5]

with mp.Pool(mp.cpu_count()) as pool:
    zipped_out = pool.map(run_experiment, rvm) # run experiment using each value of rvm

MBvals, MDvals = zip(*zipped_out)
print(MBvals)
print(MDvals)

shows

(1.0, 2.0, 3.0, 4.0, 5)                                                                                                 
(10.0, 20.0, 30.0, 40.0, 50)

edited Jun 8, 2021 at 21:36

answered Jun 8, 2021 at 21:25

David Kaftan

2,2241 gold badge14 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Bobasheto Over a year ago

Thank you for the advice. I definitely did misunderstand how pool.map works. However I noticed that this code you've given only seems to work if everything after where rvm is defined is under a "if name =='main':" including the print statements. So If I want to access the values in MBvals and MDvals does everyhting else in my code also need to be under this "if name =='main':"

David Kaftan Over a year ago

I'm sorry, I'm not able to reproduce that behavior, so I can't really help.

Gabriel Lazarini Baptistussi · Accepted Answer · 2021-06-08 20:58:25Z

0

You could try to execute the tran_for_loop function in different threads.

answered Jun 8, 2021 at 20:58

Gabriel Lazarini Baptistussi

1

Collectives™ on Stack Overflow

Parallelize a function which appends lists in python

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related