Facing Py4JJavaError on executing PySpark Code

Question

I'm trying to execute my first PySpark code in PyCharm IDE ,and facing the following exception.

from pyspark import SparkContext


def example():
    sc = SparkContext('local')
    words = sc.parallelize(["scala", "java", "hadoop", "spark", "akka"])
    print(sc.getConf().getAll())
    return words.count()


print(example())

and Printed the following data.

[('spark.master', 'local'), ('spark.rdd.compress', 'True'), ('spark.serializer.objectStreamReset', '100'), ('spark.driver.port', '59627'), ('spark.executor.id', 'driver'), ('spark.submit.deployMode', 'client'), ('spark.app.id', 'local-1526547201037'), ('spark.driver.host', 'LAPTOP-DDRRK6SB'), ('spark.ui.showConsoleProgress', 'true'), ('spark.app.name', 'pyspark-shell')]

and the following exception.

py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: java.lang.IllegalArgumentException

Sorry for my English, Expecting what's wrong with the code.

which line fails?

vvg
– vvg

2018-05-17 09:08:31 +00:00
Commented May 17, 2018 at 9:08 — vvg
– vvg, Commented May 17, 2018 at 9:08
Same works for me

pissall
– pissall

2018-05-17 09:08:46 +00:00
Commented May 17, 2018 at 9:08 — pissall
– pissall, Commented May 17, 2018 at 9:08
words.count() for executing this line facing the exception

kashyap
– kashyap

2018-05-17 09:10:05 +00:00
Commented May 17, 2018 at 9:10 — kashyap
– kashyap, Commented May 17, 2018 at 9:10
Works for me too.

vvg
– vvg

2018-05-17 09:15:30 +00:00
Commented May 17, 2018 at 9:15 — vvg
– vvg, Commented May 17, 2018 at 9:15
Which Java / JDK version do you use?

Alper t. Turker
– Alper t. Turker

2018-05-17 09:36:37 +00:00
Commented May 17, 2018 at 9:36 — Alper t. Turker
– Alper t. Turker, Commented May 17, 2018 at 9:36

kashyap · Accepted Answer · 2018-05-17 18:42:56Z

1

I don't know exact problem, On rolling back to 1.8.0_171 to this java version, It is working fine. Thanks Rumoku for your suggestion.

answered May 17, 2018 at 18:42

kashyap

5081 gold badge7 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Facing Py4JJavaError on executing PySpark Code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related