java.lang.ClassNotFoundException: Failed to find data source: org.apache.hadoop.hbase.spark when pyspark read hbase table

Question

I use pyspark to read hbase table as a dataframe, but it went some wrong:

sc = SparkContext(master="local[*]", appName="test")
spark = SparkSession(sc).builder.getOrCreate()
df = spark.read.format('org.apache.hadoop.hbase.spark') \
    .option('hbase.table', 'h_table') \
    .option('hbase.columns.mapping',
            'life_id STRING :key, score STRING info:total_score') \
    .option('hbase.use.hbase.context', False) \
    .option('hbase.config.resources', 'file:///home/softs/hbase-2.0.5/conf/hbase-site.xml') \
    .option('hbase-push.down.column.filter', False) \
    .load()

df.show()

it shows: java.lang.ClassNotFoundException: Failed to find data source: org.apache.hadoop.hbase.spark. Please find packages at http://spark.apache.org/third-party-projects.html

I followed the demo

Yayati Sule · Accepted Answer · 2020-01-12 12:31:10Z

1

The dependency is not packaged with your JAR. Use the —packages flag of spark-submit to pas the uri of the connector you are using if you don’t wish to package the dependency in your project

add the following lines to your spark-submit command:

--packages com.hortonworks:shc-core:1.1.1-2.1-s_2.11 --repositories http://repo.hortonworks.com/content/groups/public/

and it should work.

edited Jan 12, 2020 at 12:31

answered Jan 11, 2020 at 19:48

Yayati Sule

1,62914 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Yayati Sule Over a year ago

@littlely could you share your pom.xml or build.sbt file you are using for the code

littlely Over a year ago

This is pyspark, I have no porm.xml.

Yayati Sule Over a year ago

@littlely try this link. The person has found the solution for PySpark

Collectives™ on Stack Overflow

java.lang.ClassNotFoundException: Failed to find data source: org.apache.hadoop.hbase.spark when pyspark read hbase table

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related