Python parallel append to proxy list return 'function' object is not iterable or 'NoneType' object is not iterable

Question

I want to append list Python parallel processing. I write code as follow:

from joblib import Parallel
from multiprocessing.managers import BaseManager

manager = multiprocessing.Manager()
inputs = enumerate(my_list)
lproxy = manager.list()
d = lproxy
d.append([]) #init empty list

num_cores = multiprocessing.cpu_count()
_ = Parallel(n_jobs=num_cores)(d[0].append(word) for idx, word in inputs)

Then my parallel job return 'NoneType' object is not iterable. How to append to list in above case? In second example I test Parallel as follow:

from joblib import Parallel, delayed
def process(word) :
    print(word)
results = Parallel(n_jobs=num_cores)(delayed(process(word)) for idx, word in inputs)

and this return 'function' object is not iterable. Is this function have to be written somehow specially?

What's a god practice of parallel appending to the list?

ElConrado · Accepted Answer · 2019-04-09 09:17:09Z

0

It seems to be a solution:

import multiprocessing
inputs = enumerate(my_list)
manager = multiprocessing.Manager()
lproxy = manager.list()
from joblib import Parallel, delayed
def process(word) :
    lproxy.append(word)
results = Parallel(n_jobs=num_cores)(delayed(process)(word) for idx, word in inputs)

than to extract proxy list I use:

l = lproxy[:]

answered Apr 9, 2019 at 9:17

ElConrado

1,6404 gold badges23 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python parallel append to proxy list return 'function' object is not iterable or 'NoneType' object is not iterable

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related