How to use AWS lambda in python to run multiple functions at once

Question

I currently have code that gets back school data from a database and saves it to a csv file:

schoolID = '12345'

def getSchool(schoolID):
    School = SchoolsDB.find_one({"_id": ObjectId(schoolID)})
    return School
school = getSchool(schoolID)


school.to_csv(schoolID + ".csv")

It currently takes in a schoolID and runs one school at a time. I have tried putting it in a for loop so that it runs one school after another automatically, but I want to be able to run all schools at the same time.

I want to be able to use lambda to run all the schools at the same time, instead of one at a time. Does anyone know how to do this?

How does AWS Lambda factor into this? Is that code running as a Lambda function? — AKX
– AKX, Commented Mar 31, 2020 at 10:53
Have a look at: aws.amazon.com/blogs/compute/… - BTW are you using some sort Document based DB like Mongo? if you are, why not just make schoolID an array and find() all of them in one go ? — Bob
– Bob, Commented Mar 31, 2020 at 10:57
You could also create a lambda function which would distribute work to other lambda functions so kind of workers<--> manager scenario. — CodeSamurai777
– CodeSamurai777, Commented Mar 31, 2020 at 11:02
Thanks everyone for your responses. I want to make a lambda function, which sets off all the individual school functions like this one to run at the same time, but I don't know how to do that. @mrangry777 — user12913302
– user12913302, Commented Mar 31, 2020 at 11:45

Bob · Accepted Answer · 2020-03-31 11:02:52Z

1

From a purely Python POV:

It looks like you are using some form of MongoDB, rather than having the function take a single ID and executing each time, why not pass it an array and find them all in one go.

def getSchool(list_of_school_ids):
    Schooldb.collection.find( { _id : { $in : list_of_school_ids} } )

school = getSchool(list_of_school_ids=["1234", "5678"])

Then just build a massive CSV where each row is your entry in SchoolsDB - I'm coming at this completely blind though.

If not, you could look at:

https://aws.amazon.com/blogs/compute/parallel-processing-in-python-with-aws-lambda/

But my gut tells me its overkill for your use case :)

answered Mar 31, 2020 at 11:02

Bob

4656 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user12913302 Over a year ago

Thanks I do need to use lambda because my code is actually a lot more complicated with manipulating the data and other functions etc. But I just need to find an example of how to use a lambda function to trigger each individual school functions like this one, all at one.

Collectives™ on Stack Overflow

How to use AWS lambda in python to run multiple functions at once

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related