Parallelization of nested for loop

Question

My original code was -

        for j in tqdm(range(height), position = 0, leave = True):
            y = y0  + j * ystep

            for i in range(width):
                x = x0 + i * xstep
                #Do expensive task

To parallelize this I did this -

        ht_iterator = range(height)
        wt_iterator = range(width)
        paramlist = list(itertools.product(ht_iterator, wt_iterator))
        with concurrent.futures.ProcessPoolExecutor() as executor:
             executor.submit(self.process_single_pixel, paramlist)

Where self.process_single_pixel is basically the above for loop in the form of a function like so -

    def process_single_pixel(params):
        ht = params[0]; wt = params[1]
        y = y0 + ht * ystep
        x = x0 + wt * xstep
        #do expensive task 
        #expensive task calls a recursive function to implement ray-tracing

In short I was earlier iterating over the entire height and width, instead I created an iterator with all possible combinations, so that it can be individually parallelized.

Question : the program doesn’t complete execution and it seems like it gets stuck in an infinite loop. What am I doing wrong? I notice that the program never enters the function which needs to be executed and the computer hangs and becomes very slow. Also, what is the behavior for functions which do not explicitly return anything?
I tried parallelization a dummy function -

    with concurrent.futures.ProcessPoolExecutor() as executor:
          ret = executor.map(self.dummy, paramlist)

          for result in ret:
              print(result)

    def dummy(self, params):
        return (params[0], params[1])

This gave an error BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. which is apparently because these processes need to be guarded by __main__?

Sorry, forgot to add the question itself. No there was no speed up. instead my computer was lagging and CPU usage was 100% (expected, i guess?) but the output was taking way longer than it should have ( i had to force exit) — r4bb1t
– r4bb1t, Commented Jun 12, 2020 at 12:33
@PyWalker2797--you should add this comment with a question to the original post. — DarrylG
– DarrylG, Commented Jun 12, 2020 at 12:37
what is the expensive task like? Because the computation shown could be easily done with numpy in a fast way without multithreading. — Roland Deschain
– Roland Deschain, Commented Jun 12, 2020 at 13:51

Roy2012 · Accepted Answer · 2020-06-13 08:21:16Z

1

Not sure it's an answer to the question - but it might help.

This piece of code works fine, and it's basically a template for what you're looking for. Could you please try to run it, and see if it causes any issues?

import concurrent.futures
import itertools
import random
import time

class ConProc:

    def dummy(self, param):
        time.sleep(random.random() * 3) # simulate a longer job
        return param[0] * param[1]

    def main(self):
        ht_iterator = range(4)
        wt_iterator = range(5)
        paramlist = list(itertools.product(ht_iterator, wt_iterator))

        with concurrent.futures.ProcessPoolExecutor() as executor:
            ret = executor.map(self.dummy, paramlist)

            for result in ret:
                print(result)

if __name__ == '__main__':
    cp = ConProc()
    cp.main()

edited Jun 13, 2020 at 8:21

answered Jun 13, 2020 at 5:44

Roy2012

12.7k3 gold badges28 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

r4bb1t Over a year ago

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending. this was the error I got when I tried executing your code. Do you know why this might be happening?

Roy2012 Over a year ago

I added 'if name == main'. It worked fine for me without it - but can you try again on your side?

Roy2012 Over a year ago

Also - please note that you can't run it as part of an interactive console. Must be part of a py file executed with python3 my_file.py.

r4bb1t Over a year ago

Okay, the script runs. But how would I do this in the case of my function? Would I have to include an if __name__ == '__main__' guard in my main routine? Because the place where I am attempting to parallelize is inside a function, inside a class.

Roy2012 Over a year ago

Not sure I see the problem. In my example, the parallelized code is also part of a call. The 'if main' part is just a top level wrapper for the main function. What's the challenge?

|

Collectives™ on Stack Overflow

Parallelization of nested for loop

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related