Effective Python Multiprocessing in a list for loop

Question

I'm trying to multiprocess an action inside a for x in y loop. Basically, the concept of the script is to do a request to a site, load up a json file containing a list of URLs. Once fetched, another function is called to parse an URL individually. What i've been trying to do is to multiprocess this task with multiprocess.Process() in order to speed up the process since there is lots of URLs to parse. However, my approach doesn't speed up the process at all, it actually goes at the same speed than with no multiprocessing. It seems like gets blocked when using proc.join().

This is a code i've been working on:

import json
import requests
import multiprocessing

def ExtractData(id):
    print("Processing ", id)
    result = requests.get('http://example-index.com/' + id')
    result = result.text.split('\n')[:-1]
    for entry in result:
        data = json.loads(entry)['url']
        print("data is:", data)

def ParseJsonAndCall():
    url = "https://example-site.com/info.json"
    data = json.loads(requests.get(url).text)
    t = []
    for results in data:
        print("Processing ", results['url'])
        p = multiprocessing.Process(target=ExtractData, args=(results['id'],))
        t.append(p)
        p.start()
    for proc in threads:
        proc.join()

ParseJsonAndCall()

Any help would be greatly appreciated!

Didn't change anything code wise.

n00bz0r
– n00bz0r

2019-07-25 14:29:53 +00:00
Commented Jul 25, 2019 at 14:29 — n00bz0r
– n00bz0r, Commented Jul 25, 2019 at 14:29

Chen · Accepted Answer · 2019-07-25 15:34:41Z

1

A Pool may help.

import multiprocessing as mp    

def ParseJsonAndCall():
    url = "https://example-site.com/info.json"
    data = json.loads(requests.get(url).text)
    collect_results = []
    with mp.Pool(processes=mp.cpu_count()) as pool:
        for results in data:
            res = pool.apply_async(ExtractData, [results['id'],])
            collect_results.append(res)
        for res in collect_results:
            res.get()

Although the print statement in ExtractData() might cause a race condition.

edited Jul 25, 2019 at 15:34

answered Jul 25, 2019 at 14:07

Chen

3303 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

n00bz0r Over a year ago

Thanks. I'm not sure how to implement this with the code I provided. Also, I use Process(), would that be compatible with it?

n00bz0r Over a year ago

With Pool.apply_async the call returns immediately instead of waiting for the result, therefore ExtractData is never completed (add a print('Job completed') at the end of the function). Maybe there's need for a lock?

Chen Over a year ago

Ah, my mistake, forgot to call get().

Collectives™ on Stack Overflow

Effective Python Multiprocessing in a list for loop

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related