Data Processing in the Work of NoSQL? An Introduction to Hadoop

Data Processing in NoSQL?

An Introduction to Map Reduce
By Dan Harvey

Thursday, 12 April 12

No SQL?

People thinking about
their data storage.


Storage Patterns

Denormalization
Sharding / Hashing
Replication
...


Ad Hoc Queries?

Hard to do...


The problem
High

In-memory Hadoop
Query Entropy

(Offline)

Key-value ?
Low
(Online)

Low High

Query Latency


“The Apache Hadoop project
develops open-source software
for reliable, scalable,
distributed computing”


MapReduce: Simpliﬁed Data Processing on Large Clusters

Jeffrey Dean and Sanjay Ghemawat
jeff@google.com, sanjay@google.com

Google, Inc.

Abstract given day, etc. Most such computations are conceptu-
ally straightforward. However, the input data is usually
MapReduce is a programming model and an associ- large and the computations have to be distributed across
ated implementation for processing and generating large hundreds or thousands of machines in order to ﬁnish in
data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par-
key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle
pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu-
values associated with the same intermediate key. Many tation with large amounts of complex code to deal with
real world tasks are expressible in this model, as shown these issues.
in the paper. As a reaction to this complexity, we designed a new
Programs written in this functional style are automati- abstraction that allows us to express the simple computa-
cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de-
modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution

Google’s MapReduce
details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in-
gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp
chine failures, and managing the required inter-machine and many other functional languages. We realized that
communication. This allows programmers without any most of our computations involved applying a map op-
experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to
ily utilize the resources of a large distributed system. compute a set of intermediate key/value pairs, and then
Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared
cluster of commodity machines and is highly scalable: the same key, in order to combine the derived data ap-
Thursday, 12 April 12 propriately. Our use of a functional model with user-

Distributed Storage


Google, Inc.
Abstract given day, etc. Most such computations are conceptu-
Google, Inc. ally straightforward. However, the input data is usually
Jeffrey Dean and Sanjay Ghemawatmodel and an associ-
MapReduce is a programming large and the computations have to be distributed across
jeff@google.com, sanjay@google.com function that processes a
data sets. Users specify a map
MapReduce: Simplified Data Processing on Large Clusters key/value pair to generate a set of intermediate key/value
a reasonable amount of time. The issues of how to par-
allelize the computation, distribute the data, and handle
Abstract a reduce function that merges all intermediate such computations are conceptu-
pairs, Google, Inc.
and given day, etc. Most
failures conspire to obscure the original simple compu-
values associated with the same intermediate key. ManyHowever, the input data is usually
MapReduce is a programming model and an associ-
ally straightforward.
tation with large amounts of complex code to deal with
Jeffrey Dean and Sanjay Ghemawat real world tasks are expressible largemodel, computations have to be distributed across
this and the
ated implementation for processing and generating large in hundreds or as shown of these issues. order to finish in
thousands machines in
data sets. Users specify a in thefunction that processes a
map paper. a reasonable amount of time. Thereaction to thisto par-
As a issues of how complexity, we designed a new
jeff@google.com, sanjay@google.com Abstract Programs written key/value Most such are automati- areabstraction that allows
given day, etc. computations conceptu-
key/value pair to generate a set of intermediate in this functional stylethe computation, distribute the data, and us to express the simple computa-
allelize handle
ally straightforward.large clusterthe com- data is usually
However, of input
Google, Inc. MapReduce is apairs, and a reduce function that merges all intermediate on a
programming model and cally parallelized andand the computations conspirebe distributed across trying to perform but hides the messy de-
an associ- large
executed
failures have to to obscure thewe were simple compu-
tions original
values associated with the modity machines. The run-time system takes care of the
same intermediate key. Many
ated implementation for processing and generating large tation with large amounts tails of parallelization, fault-tolerance, data distribution
of finish in code to deal with
complex
hundreds or thousands of machines in order to
data sets. Users specifyworld tasks are expressible in this model, as shown data, scheduling the pro- of how to par-
real a map function that processes partitioning the input these issues.
details of a and load balancing in a library. Our abstraction is in-
gram’s execution reasonable amount of time. The issues
key/value pair to generatepaper.of intermediate key/value
in the a set
a across a set of machines, handling ma-
As distribute to data, and handle map and reduce primitives present in Lisp
spired by the
allelize the computation, a reaction thethis complexity, we designed a new
chine failures, and managing the required inter-machine
pairs, and a reduce function that merges all intermediate style are automati- toabstraction that allowssimple compu-other functional languages. We realized that
and manythe simple computa-
Abstract given day, etc. Most such computations are conceptu- Programs written in this functional conspire obscure the original us to express
failuresallows

Replicated Blocks
communication. This of com- programmers without any
values associated with the same intermediate key. on a large cluster with large amounts of complex code to deal but hides the messy involved applying a map op-
cally parallelized and executed Many most of our computations de-
ally straightforward. However, the input data is usually tation tions we were trying to perform with
MapReduce is a programming model and an associ- large and the computations have to be distributed across real world tasks are expressible in this model, as shown withthese issues. distributedof parallelization, fault-tolerance, data distribution in our input in order to
experience parallel and
modity machines. The run-time system takes care of the tails
systems to eas- eration to each logical “record”
ated implementation for processing and generating large in the paper. ily utilize the resources ofpro-
a large distributed system.
details of partitioning the input data, scheduling a reaction to and load balancing in a library. new set of intermediate key/value pairs, and then
compute a
hundreds or thousands of machines in order to finish in As the this complexity, we designed a Our abstraction is in-
data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par- gram’s functional style a setOurmachines, handling ma- allows usrunsthe map and reduce primitives present in Lisp all the values that shared
of implementation of that
Programs written in thisexecution across are automati- abstraction spired to express the simple computa- reduce operation to
MapReduce by on a large applying a
key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle cally parallelized and executed on and managing of com-commodity machines and to many other functional messy de- on Largethat
chine failures, a large cluster the requiredtions we were trying is performscalable:Processing We realized combine the derived data ap-
cluster of MapReduce: Simplified Data
inter-machine and highly but hides thethe same key, in order to Clusters
languages.
pairs, and a reduce function that merges all intermediate failures conspire to obscure the original simple compu- communication. This allows programmers without any
care of the most of our computations propriately. Our use mapaop-
modity machines. The run-time system takes a typical MapReduce computation processes many ter- involved applying a of functional model with user-
tails of parallelization, fault-tolerance, data distribution
values associated with the same intermediate key. Many tation with large amounts of complex code to deal with experience data, scheduling distributed on thousands of machines. Programmers “record” in map and in order to
abytes of data
details of partitioning the inputwith parallel and the pro- systems to balancingeration to each Our abstraction is in-
and load eas- in a library. logical
specified our input reduce operations allows us to paral-
real world tasks are expressible in this model, as shown these issues. a large system easy to by the map Jeffrey Dean and Sanjay in large computations then and to use re-execution
gram’s execution across a set the machines, of find thema- spired
use: MapReduce intermediate key/value pairs, and easily
ily utilize of resources handling distributed system. hundreds ofreduce set of pro- present Ghemawat
compute a primitives
and
lelize
Lisp
in the paper. As a reaction to this complexity, we designed a new MapReduce: Simplified Data Processing one thou- as the primary
grams have been implemented and applying a on Large Clustersvalues that shared fault tolerance.
upwards ofreduce operation to all the mechanism for
chine failures, and managing the required inter-machine runs on a large functional languages. We realized that
Our implementation of MapReduce and many other
Programs written in this functional style are automati- abstraction that allows us to express the simple computa- sand MapReduce jobs are executed onjeff@google.com, sanjay@google.com contributions ap-this work are a simple and
cluster of programmers without any Google’s clusters to combine the derived data of
the same key, in order
communication. This allows commodity machines and is highly of our computations involved applying a map op- The major
most scalable:
cally parallelized and executed on a large cluster of com- tions we were trying to perform but hides the messy de- experience with parallel andMapReduce systems today.processes many each logical “record” Our use of apowerfulto model that enables automatic parallelization
every eas-
a typical distributed computation eration to ter- propriately. in our input in order interface with user-
functional
modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution ily utilize the resources of oflarge distributed system. specified mapGoogle, Inc. distribution of large-scale computations, combined
and
abytes a data on thousands of machines.Dean and set of intermediate key/value pairs, and then allows us to paral-
Jeffrey compute a Sanjay Ghemawatand reduce with an implementation of this interface that achieves
Programmers operations
details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in- Our implementation the system easy to use:1hundreds of MapReduceapro-
find of MapReduce runs on a large lelize large computations easily and to use re-execution
applying reduce operation to all the values that shared
Introduction
gram’s execution across a set of machines, handling ma- spired by the map and reduce primitives present in Lisp MapReduce: Simplified scalable: the one key, Large Clusters derived Section 2 describeslargebasic programming model and
cluster of commodity machines and is highlyData upwards ofsame thou- in order toprimary mechanism for fault tolerance.
grams have been implemented and Processing on as the combine the
high performance on
data ap-
the
clusters of commodity PCs.
chine failures, and managing the required inter-machine and many other functional languages. We realized that sand MapReduceprocessesexecuted on Google’s clusters useThe a functional model with user- are a simple and
a typical MapReduce computation jobs are many ter- propriately. Our of major contributions of this work
communication. This allows programmers without any most of our computations involved applying a map op- every day.
abytes of data on thousands of machines. Programmers AbstractInc.
Google,
Over the past five years, mapauthors and many othersthat enables paral- Most such computations are conceptu-
specified the and powerful interface allows us to automatic parallelization
reduce operations at given day, etc. examples. Section 3 describes an imple-
gives several
Google have implemented computationsspecial-purpose ally straightforward.MapReduce interface tailored towards
lelize large hundreds of easily andof use re-executionof the combined
mentation
experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to find the system easy to use: hundreds of MapReduce pro- is a programming model and an associ- large-scale computations,However, the input data is usually
and distribution to
Jeffrey Dean and Sanjay Ghemawat mechanism for fault tolerance.ourand interface that achieves to be distributed acrossde-
MapReduce that process large amounts of raw data, large cluster-based computing environment. Section 4
ily utilize the resources of a large distributed system. compute a set of intermediate key/value pairs, and then computations as the primary with an implementation of this the computations have
grams have been implemented and upwards of one thou-
Our implementation of MapReduce runs on a large applying a reduce operation to all the values that shared 1 Introduction ated implementationdocuments, contributions of thisetc.,on largescribes and
such as crawled for processing and generating large to hundreds several refinements of the programming model
jeff@google.com,Users The map
web high performance
request logs,
sand MapReduce jobs are executed on Google’s clusters specify amajor function that processes a are a simple orof commodity machines in order to finish in
work clusters thousands of PCs.
data sets. sanjay@google.comderived data, such as inverted a reasonable amount ofuseful. The issues of how to par-
compute various kinds of
cluster of commodity machines and is highly scalable: the same key, in order to combine the derived data ap- every day. Abstract given day, etc. 2 describes parallelization are conceptu- Section 5 has performance
Section automatic computations found time.
that we have
powerful interface that enables Most suchthe basic programming model and
key/value pair to generate a set of intermediate key/value
a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user- Over the past five years, the authors and many others at allyoflarge-scale computations,measurements of usually
indices, various representations gives severalstructure allelize inputdescribes an distribute the data, and variety of
the graph examples. Section
Google, Inc. summaries of straightforward. However,tasks.the3computation, imple-
combineddata is
the our implementation for a handle
pairs, and documents, distribution of
and
MapReduce is a programming web a reduce function that merges allthe computations have tointerface tailored towardsoriginalMapReduce within
intermediate
abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral- Google have implementedof model andspecial-purpose largethe number of pages failures beSection to explores the use of simple compu-
hundreds of an associ-
with an implementation of this the MapReduce conspire obscure
and
mentation of interface that achieves distributed across the
6
lelize large computations easily and to use re-execution 1 Introduction ated computations that process large amounts of the same most our cluster-based computing environment. Section 4 complex code toitdeal with
implementation for processingassociated with raw data,intermediate key. Many of machines in includingfinish in de-
values andper host, the set of
crawled generating large frequentthousands a tation with order amounts of
hundreds or queries in Google large to our experiences in using as the basis
find the system easy to use: hundreds of MapReduce pro- high performance on large clusters of commodity PCs.
as the primary mechanism for fault tolerance. data such as crawled documents, web tasks arelogs, etc., toin reasonableseveral refinements of issues of how to par-
sets. Users specify a map functionrequest expressible a this model,amount of time. The the programming model
real world that processes a
scribes
as shown these issues.
grams have been implemented and upwards of one thou- Section 2 describes the basic programming model and
key/value pairvarious kinds setthe intermediate key/value
to generate a in derived data, such as invertedallelize the computation, useful. Section 5 has this complexity, we designed a new
of paper.
sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and Over the past five Abstractauthors and many others at given day,several examples.we have found distribute imple- to performance
compute
years, the of that Section 3 describes ana the
As reaction
gives etc. Most such computations are conceptu- data, and handle
pairs, and a reduce function that merges written in this functional measurements of our implementation allows variety of the simple computa-
Programs all intermediate style are automati-
every day. powerful interface that enables automatic parallelization Google have implemented hundredsrepresentations of the graph structure failures conspire on Operating usually that for andus to express
of MapReduce abstraction
indices, various of special-purpose ally mentation ’04:the However, theinterface tailoredoriginal simple a Implementation
straightforward.a large cluster ofto obscureisthe towards Design compu-
values associated with and USENIX Association executed on6th Symposium inputexplores Systemsof MapReduce withinbut hides the messy de- 137
the same intermediate and OSDI
cally parallelized key. Many
MapReduce is athat of web documents, summaries of the number the pages computing to com- data
large be distributed thewe 4 de- trying to with
tation with Section 6amounts ofSection were to deal perform
tions use code
complex
and distribution of large-scale computations, combined computations programming model an associ-
process large amounts of raw data, and of computations have environment. across
our cluster-based tasks.
with an implementation of this interface that achieves real crawled per and generating in this largeThe run-time
ated such as crawled for processing are expressible large model,scribes severalthese issues. including our totails of in in using it as fault-tolerance, data distribution
world tasks host, the set of most machines. as shown
implementation documents, web request modityetc., frequent queries in asystem takes carethe order experiences
Google of in theof parallelization, the basis
logs, to hundreds or thousands of machines
refinements programming model
finish
1 Introduction high performance on large clusters of commodity PCs. data compute various theapaper.derived data, such as inverteda reasonable have data, schedulingSection 5 complexity, we designed aanew
in
sets. Users specify map function that processesof partitioning the input found time. The the pro- hasand load balancing in library. Our abstraction is in-
kinds of
details a As a reaction to this
that we amount of useful. issues of how to par- performance
Section 2 describes the basic programming model and key/value pair to generate a set of intermediatefunctional style are measurementsmachines, handling ma- forspired by the map and reduce primitives present in Lisp
indices, various representations of
gram’s execution across a set of abstraction that allows us to express the simple computa-
Programs written inthe graph structure
this key/value automati-
allelize the computation, distribute the data, anda handle of
of our implementation variety
Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple- pairs, and a reduce cally parallelized of the number of large cluster of com- the tions we wereoriginalto perform many other functional languages. We realized that
function that merges executed on failures,failures conspire to required inter-machine
of web documents, summaries and all intermediate
chine a
pages
and managing and but hides
tasks. Section 6 explores thetrying simple compu-
obscure the use of MapReduce within the messy de-
Google have implemented hundreds of special-purpose mentation of the MapReduce interface tailored towards values associated host, the same intermediate key.’04: 6thin takes This of the programmerscomplex inand to most with data distribution
modity set of most frequent queries Symposium allows tails Systems Design using it as of basis
USENIX Association OSDI system communication. care on Operating of parallelization, fault-tolerance,
without any Implementation
crawled per with the machines. The run-timeMany a tation with large amounts experiences code deal the our computations involved applying a map op-
Google including our of 137
computations that process large amounts of raw data, our cluster-based computing environment. Section 4 de- real world tasks are expressible in this model, experience with parallelpro- distributed systems to eas-a library. Our each logical is in-
details of partitioning the inputshownscheduling the and
as data, these issues. a largeand load balancing in
eration to abstraction “record” in our input in order to
in the paper. gram’s execution across a set of machines, handlingof
ily utilize the resources ma- distributed system. compute a set of intermediate key/value pairs, and then
such as crawled documents, web request logs, etc., to scribes several refinements of the programming model As a reaction to spired by the mapwe designed primitives present in Lisp
this complexity, and reduce a new
compute various kinds of derived data, such as inverted that we have found useful. Section 5 has performance chine failures, and managing the required inter-machine
Programs written in this functional style are automati- and many other the simple computa-a reduce operation to all the values that shared
Our implementation of MapReduce runs on a large applying
abstraction that allows us to express functional languages. We realized that
communication. This allows programmers without any the same key, 137in order to combine the derived data ap-
indices, various representations of the graph structure measurements of our implementation for a variety of cally parallelized and executed’04:a6th Symposium com- commodity machines mostis Implementation the involved applying a map op-
USENIX Association OSDI on large cluster of on of cluster Operating Systems Design andofhighly scalable:
and
tions we were trying to performcomputations messy de-
our but hides
of web documents, summaries of the number of pages tasks. Section 6 explores the use of MapReduce within experience with parallel and distributed systems computation processes many ter- “record” in our input in orderato
modity machines. The run-time system takes care of the MapReduceto eas-
a typical eration to each logical propriately. Our use of functional model with user-
tails of parallelization, fault-tolerance, data distribution
ily utilize the resources of a abytesdistributed thousands of machines. a set of intermediate key/value pairs, reduce operations allows us to paral-
details of partitioning the input data, scheduling the pro- dataandsystem.
large of on Programmers specified map and and then
crawled per host, the set of most frequent queries in a Google including our experiences in using it as the basis load balancingcompute
in a library. Our abstraction is in-
find ma- applying a primitives present in Lisp computations easily and to use re-execution
spired a the map and reduce reduce
Our implementation of MapReduce runs on by large pro- lelize large
gram’s execution across a set of machines, handlingthe system easy to use: hundreds of MapReduceoperation to all the values that shared
as the primary mechanism for fault tolerance.
chine failures, and cluster of commodity machines and is been implemented functional languages. Weto combine the derived data ap-
managing the required inter-machine highlymany other and upwards of one order realized that
grams have
and scalable: the same key, in thou-
communication. This allows programmers without MapReduce jobs our executed on Google’s clusters a afunctional model with user- this work are a simple and
sand any
a typical MapReduce computation processes many ter-
most of
are propriately. Our use of
computations involved applying map op-
The major contributions of
USENIX Association OSDI ’04: 6th Symposium on Operating Systems Design and Implementation 137 of machines. eration to each logical “record” in our input operations interface thatparal- automatic parallelization
specified map and reduce in order to allows us to enables
powerful
experience with parallel and data on thousandseveryeas-
abytes of distributed systems to day. Programmers
ily utilize the resources thea large distributed system. of MapReduceapro- of intermediate computations easily distributionre-execution computations, combined
find of system easy to use: hundreds lelize large key/value pairs, and and to use of large-scale
Thursday, 12 April 12 Our implementation ofhave been implementedaand upwards of one thou-
grams MapReduce runs on large
compute set and then
with an implementation of this interface that achieves
applying a reduce operation to allmechanismthat fault tolerance.
as the primary the values for shared

Map, Shufﬂe, Reduce


Data Locality: Map


Google, Inc.
MapReduce, Simplified, Data, Processing,
Large, Clusters, Jeffrey, Dean, and, Sanjay,
Google, Inc.
Ghemawat, Google, Inc, ...
Abstract Google, Inc. given day, etc. Most such computations are conceptu-
ally straightforward. However, the input data is usually
MapReduce is a programming model and an associ- large and the computations have to be distributed across
data sets. Users specify a map function that processes a a reasonable amount of time. The issues of how to par-
key/value pair to generate a set of intermediate key/value allelize the computation, distribute the data, and handle
pairs, and a reduce function that merges all intermediate

Map Stage
failures conspire to obscure the original simple compu-
values associated with the same intermediate key. Many
MapReduce: Simplified Dataof complex code to deal with
real world tasks are expressible in this model, as shown
tation with large amounts Processing on Large Clusters
these issues.
in the paper. As a reaction to this complexity, we designed a new
Programs written in this functional style are automati- abstraction that allows us to express the simple computa-
cally parallelized and executed on a large cluster of com- Jeffrey Dean and Sanjay Ghemawat
tions we were trying to perform but hides the messy de-
modity machines. The run-time system takes care of the tails of parallelization, fault-tolerance, data distribution
details of partitioning the input data, scheduling the pro- and load balancing in a library. Our abstraction is in-
gram’s execution across a set of machines, handling ma- spired by the map and reduce Inc.
Google, primitives present in Lisp
chine failures, and managing the required inter-machine and many other functional languages. We realized that
communication. This allows programmers without any most of our computations involved applying a map op-
experience with parallel and distributed systems to eas- eration to each logical “record” in our input in order to
ily utilize the resources of a large distributed system. compute a set of intermediate key/value etc. Most then computations are conceptu-
Abstract given day, pairs, and such
Our implementation of MapReduce runs on a large applying a reduce operation to all straightforward.shared
ally the values that However, the input data is usually
cluster of commodity machinesMapReduce is a programming model key, an order to combineand the computations have to be distributed across
and is highly scalable: the same and in associ- large the derived data ap-
ated implementation for processing and generating large a functional model with user-machines in order to finish in
a typical MapReduce computation processes many ter- propriately. Our use of hundreds or thousands of
abytes of data on thousands of machines. Programmersmap specified that etc. Most a operations allows are to paral- The issues of how to par-
Abstract Users specify a
data sets. function mapprocesses
and reduce us

MapReduce, programming, model, associ-
given day, such a reasonable amount of time.
computations conceptu-
find the system easy to use: hundreds of MapReduce pro- set ofally straightforward. However,and to use data is usually
key/value pair to generate a lelize large computations easily
intermediate key/value allelize input re-execution
the the computation, distribute the data, and handle
grams have been is a programmingupwardsandone associ-that merges allthe mechanism forhave to be distributed across the original simple compu-
MapReduce implemented and and a reduce function aslargeprimary computations failures conspire to obscure
pairs, model of an thou- the
and
intermediate fault tolerance.
sand MapReduce jobs for processingassociated with the same intermediatecontributions of tation withare a simple andin complex code to deal with
values and generating large
ated implementation are executed on Google’s clusters The major key. Many this work large
hundreds or thousands of machines in order amounts of
to finish

ated, implementation, processing,
every day. Users specify a map function that are expressiblepowerful interface shown time. automatic parallelization
data sets. real world tasks processes a in this model, as that enables
a reasonable amount of these issues. of how to par-
The issues
in the intermediate key/value and distribution of large-scale computations, combined
key/value pair to generate a set of paper.
Abstract allelize the computation, distribute the data, to this complexity, we designed a new
As a reaction and handle
pairs, and a reduce function that merges written day, etc. Most suchconspire to obscureabstraction that allows us to express the simple computa-
Programs all intermediate with an implementation are this interface that achieves
given in this functional stylecomputations of conceptu-
failures are automati-
1 Introduction the cally parallelized key. Many highHowever, the on large clustersoriginal simple PCs.
ally straightforward. performance input data isthe of commodity compu-
usually
apReduce is avalues associated with and same intermediate and executed on a large cluster ofamounts of complex code to deal with
programming model an associ- tation with large com- tions we were trying to perform but hides the messy de-

generating, large, data, sets, Users, ...
real world tasks are expressible large model, The run-time system takes carebe distributedprogramming model and
mplementation for processing and generating modity this large as shown
in machines. and the computations have to of the basic across
Section 2 describes the
these issues. tails of parallelization, fault-tolerance, data distribution
Over the past five years, the authors and many othersor thousands several examples. Section load balancing imple-
hundreds at gives of machines in order toand 3 describes an in library. Our abstraction is in-
finish in
sets. Users specify apaper. function that processesof partitioning the input data, scheduling the pro- complexity, we designed aanew
in the map details a As a reactionissues of
to this
Google have implemented hundreds execution across a set mentation of The MapReduce interface tailored towards
gram’s of a reasonable amount of time. that allows us how to par- simple computa-
Programs written in this key/value special-purpose the
alue pair to generate a set of intermediatefunctional style are automati- of machines, handling ma- to spired by the map and reduce primitives present in Lisp
abstraction express the
allelize the computation, distributecomputing environment. Section 4 de-
the data, and handle
computations that process large amounts of raw data, the required inter-machine
chine failures, and managing our cluster-based and many other functional languages. We realized that
and a reduce function that merges all intermediate
such as crawled documents, communication. This conspire scribes several original simple compu- computations involved applying a map op-
failures allows to obscure the refinements
web request logs, etc., to programmers without any of the of our most programming model
s associated with the same intermediate key. Many tation with and amounts of complex eas- to deal with
large that we systems to code
compute various kinds of derived data, such as inverted distributedhave found useful. Section to has performance
eration 5 each logical “record” in our input in order to
world tasks are expressible in this model, experience with parallel
as shown these issues.
indices, various representations utilize the resources of a large distributed system. implementation afor a variety of
ily of the graph structure measurements of our compute set of intermediate key/value pairs, and then
paper.
of web documents, summaries of the number ofapages
As reaction to this complexity, we designed a of MapReduce within
use new
tasks. Section 6 explores the applying a reduce operation to all the values that shared
ograms written in thisper host, the set are most frequent abstraction that allows us including athe experiences in using it as the basis
Our implementation of MapReduce runs on large
crawled functional style of automati- queries in a Google to expressour simple computa- key, in order to combine the derived data ap-
cluster of commodity machines and is highly scalable: the same
a typical MapReduce computation processes many ter- propriately. Our use of a functional model with user-
abytes of data on thousands of machines. Programmers specified map and reduce operations allows us to paral-
lelize large computations easily and to use re-execution
USENIX Association OSDI ’04:find the system easy to use: hundreds of MapReduce pro-
6th Symposium on Operating Systems Design and Implementation 137
grams have been implemented and upwards of one thou- as the primary mechanism for fault tolerance.
sand MapReduce jobs are executed on Google’s clusters The major contributions of this work are a simple and
every day. powerful interface that enables automatic parallelization
and distribution of large-scale computations, combined
with an implementation of this interface that achieves
1 Introduction high performance on large clusters of commodity PCs.
Section 2 describes the basic programming model and
Over the past five years, the authors and many others at gives several examples. Section 3 describes an imple-
Google have implemented hundreds of special-purpose mentation of the MapReduce interface tailored towards
Thursday, 12 April 12 computations that process large amounts of raw data, our cluster-based computing environment. Section 4 de-

Shuffle
and, associated, clusters, data, data, dean,
MapReduce, Simplified, Data, Processing,
implementation, inc, jeffrey, large, large,
Large, Clusters, Jeffrey, Dean, and, Sanjay,
mapreduce, mapreduce, model, generating,
Ghemawat, Google, Inc, ...
ghemawat, google, google, ...

MapReduce, programming, model,
programming, processing, processing,
associated, implementation, processing,
sanjay, simplified, sets, users, ...
generating, large, data, sets, Users, ...


Reduce
and, 1
associated, 1
clusters, 1
and, associated, clusters, data, data, dean, data, 2
implementation, inc, jeffrey, large, large, ...
mapreduce, mapreduce, model, generating, mapreduce, 2
ghemawat, google, google, ... model, 1,
generating, 1
ghemawat, 1
google, 2

programming, 1
processing, 2
programming, processing, processing,
sanjay, 1
sanjay, simpliﬁed, sets, users, ...
...


So Hadoop?

• Map Reduce for processing
• Distributed File System (HDFS) for storage


Other projects


Uses
• Log analysis
• Data Warehouse
• Natural Language Processing
• Search indexes
• Machine Learning
• ...

1.0 and beyond

• Next generation Hadoop - YARN
• Not just MapReduce
• General resource allocation framework


Users


Want more?

www.meetup.com/Hadoop-Users-Group-UK/

(April 25th)

BigDataWeek.com

(April 23rd - 29th)


Data Processing in the Work of NoSQL? An Introduction to Hadoop

More Related Content

What's hot

Similar to Data Processing in the Work of NoSQL? An Introduction to Hadoop

Recently uploaded

Data Processing in the Work of NoSQL? An Introduction to Hadoop

Editor's Notes