How does MapReduce framework implement the sort phase?

Question

I am interested in the implementation of the MapReduce sort phase; it seems to be very efficient. Could someone provide some references about it please? Thanks!

Tejas Patil · Accepted Answer · 2012-02-21 01:58:50Z

7

This points to ReduceTask.java as the place where sort phase is coded. See lines 393-408 in ReduceTask.java. If you need more info, download the entire source and dig into it.

EDITED

"Sort" phase falls under ReduceTask as shown in this figure below from hadoop book. (Page no: 163) enter image description here

edited Feb 21, 2012 at 1:58

answered Feb 20, 2012 at 18:42

Tejas Patil

6,1892 gold badges26 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Thomas Jungblut Over a year ago

Attention: The sort happens actually after the mapper. The reducer just merges the sorted segments.

Tejas Patil Over a year ago

@ThomasJungblut: see the figure above.

Tejas Patil Over a year ago

@ThomasJungblut sorting happens over the output of map() method and the inputs that a reducer gets from several mappers. The later is termed as the "Sort Phase" (as shown in the pic).

Thomas Jungblut Over a year ago

the sorting is parallel to the output to map, still the sort phase is just a merge phase. So the naming is misleading. That is what my intention was (a few months ago).

Collectives™ on Stack Overflow

How does MapReduce framework implement the sort phase?

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related