How to sort Array[Row] by given column index in Scala

Question

How to sort Array[Row] by given column index in Scala?

I'm using RDD[Row].collect() which gives me array[Row], but I want to sort it based on a given column index.

I have already used quick-sort logic and it's working, but there are too many for loops and all.

I would like to use a Scala built-in API which can do this task with the minimum amount of code.

Tzach Zohar · Accepted Answer · 2016-08-03 09:25:22Z

1

It would be much more efficient to sort the Dataframe before collecting it - if you collect it, you lose the distributed (and parallel) computation. You can use Dataframe's sort, for example - ascending order by column "col1":

val sorted = dataframe.sort(asc("col1"))

answered Aug 3, 2016 at 9:25

Tzach Zohar

37.9k3 gold badges83 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

The Archetypal Paul Over a year ago

"It would be much more efficient to sort the Dataframe before collecting it " If it fits on one node (i.e. can be collected) I'll bet it's more efficient to sort after collect :) but in general it won't be possible to collect an RDD, or one wouldn't be using Spark...

Collectives™ on Stack Overflow

How to sort Array[Row] by given column index in Scala

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related