Unable to import module 'lambda_function': No module named 'pyspark'

Question

I'm trying to run my PySpark job code using AWS Lambda function. Here i'm getting below error: Unable to import module 'lambda_function': No module named 'pyspark'

My PySpark job has below imports. So, how can I import below library into my Lambda function. How to install these in Lambda function ?

from pyspark.sql import SparkSession
from pyspark.sql import functions as F, Column as col
from pyspark.sql.functions import when, coalesce
from pyspark.sql.functions import lit

Thanks.

adiga · Accepted Answer · 2019-06-24 12:10:29Z

1

Guys you must aware three things, when you got this error

Unable to import module 'lambda_function'

File permission
handler format would be python_filename.lambda_handler
When you compress files you must give option as zip -r name.zip. (hidden files are add only when we end dot(.) at end of zip file)

edited Jun 24, 2019 at 12:10

adiga

35.4k9 gold badges65 silver badges88 bronze badges

answered Jun 24, 2019 at 12:08

Gokul D

413 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

yuriy polonskiy · Accepted Answer · 2018-12-11 09:49:53Z

0

You need to package all AWS Lambda dependencies together with your code into deployment zip file.

https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

answered Dec 11, 2018 at 9:49

yuriy polonskiy

312 bronze badges

Collectives™ on Stack Overflow

Unable to import module 'lambda_function': No module named 'pyspark'

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related