Spark.sql and sqlContext.sql

Question

I have imported the below modules. I tried to load data from sqlCtx.read.format, I am getting "IllegalArgumentException: u"Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':"" error, but it works well when I use spark.read.format. I am seeing same behavior when I am retrieving data from registered temptable/view. What can I add extra to use sqlCtx.sql instead of spark.sql?

import os
import sys
import pandas as pd
import odbc as pyodbc
import os
import sys
import re
from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
from pyspark.sql import Row
from pyspark.sql.functions import *
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import pyspark.sql.functions as func
import matplotlib.patches as mpatches
import time as time
from matplotlib.patches import Rectangle
import datetime
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
conf = SparkConf()
conf.setMaster("local")
conf.setAppName("AppName")
sqlCtx = SQLContext(sc)

Which version of Spark do you use? Maybe it will help others to solve this problem. — titipata
– titipata, Commented Mar 15, 2017 at 23:40

nick · Accepted Answer · 2019-06-26 15:59:45Z

11

I spent two hours of my life in this one, just to realize I did not need:

sqlCtx = SQLContext(sc)

Just using SQLContext.read.(...), solved this in my case.

edited Jun 26, 2019 at 15:59

nick

1,1602 gold badges11 silver badges26 bronze badges

answered Mar 31, 2017 at 4:02

Kieleth

5065 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark.sql and sqlContext.sql

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related