filter DataFrame with Regex with Spark in Scala

Question

I want to filter out rows in Spark DataFrame that have Email column that look like real, here's what I tried:

df.filter($"Email" match {case ".*@.*".r => true case _ => false})

But this doesn't work. What is the right way to do it?

Use rlike like described here: stackoverflow.com/questions/27249685/… — TomTom101
– TomTom101, Commented Nov 27, 2015 at 21:53

Matthew Gray · Accepted Answer · 2015-11-27 22:05:53Z

37

To expand on @TomTom101's comment, the code you're looking for is:

df.filter($"Email" rlike ".*@.*")

The primary reason why the match doesn't work is because DataFrame has two filter functions which take either a String or a Column. This is unlike RDD with one filter that takes a function from T to Boolean.

answered Nov 27, 2015 at 22:05

Matthew Gray

1,67616 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Srini Over a year ago

Matthew, this bit of code works in the Spark REPL but not in my main file. What import do I need to run this?

Matthew Gray Over a year ago

@BruceWayne, the REPL configures some things for you that you need to configure yourself in an App. So two Qs: 1) Does your function work with simpler operations (you can successfully create and count a dataframe, for example)? 2) are you running the same Spark version for your REPL and wherever the main file is run?

Srini Over a year ago

Matthew, you're right. I needed to add val sqlContext= new org.apache.spark.sql.SQLContext(sc) and import sqlContext.implicits._

Collectives™ on Stack Overflow

filter DataFrame with Regex with Spark in Scala

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related