Downloaded 31 times
![high_temp=0forline inopen('1901'): line =line.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if(temp !="+9999"and quality in"01459"): high_temp=max(high_temp,float(temp)) printhigh_tempHow can we make this scale?( and do more interesting things )](https://image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-3-2048.jpg)
![defmapper(line): line =line.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if(temp !="+9999"and quality in"01459"): returnfloat(temp) returnNoneoutput =map(mapper,open('1901')) printreduce(max,output)Mapreduce in pure python](https://image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-7-2048.jpg)
![for line in sys.stdin:val = line.strip() (year, temp, q) = (val[15:19], val[87:92], val[92:93]) if (temp != "+9999" and re.match("[01459]", q)): print "%s\t%s" % (year, temp)mapper.py(last_key, max_val) = (None, 0)for line in sys.stdin: (key, val) = line.strip().split("\t") if last_key and last_key != key: print "%s\t%s" % (last_key, max_val) (last_key, max_val) = (key, int(val)) else: (last_key, max_val) = (key, max(max_val, int(val)))if last_key: print "%s\t%s" % (last_key, max_val)reduer.pycat dataFile | mapper.py | sort | reducer.pyHadoop Streaming](https://image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-8-2048.jpg)
![def mapper(key,value): line = value.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if (temp != "+9999" and quality in "01459"): yield year, int(temp)def reducer(key,values): yield key,max(values)if __name__ == "__main__": import dumbodumbo.run(mapper,reducer,reducer)Dumbo](https://image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-10-2048.jpg)
![high_temp=0forline inopen('1901'): line =line.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if(temp !="+9999"and quality in"01459"): high_temp=max(high_temp,float(temp)) printhigh_tempHow can we make this scale?( and do more interesting things )](https://crownmelresort.com/image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-3-2048.jpg)
![defmapper(line): line =line.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if(temp !="+9999"and quality in"01459"): returnfloat(temp) returnNoneoutput =map(mapper,open('1901')) printreduce(max,output)Mapreduce in pure python](https://crownmelresort.com/image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-7-2048.jpg)
![for line in sys.stdin:val = line.strip() (year, temp, q) = (val[15:19], val[87:92], val[92:93]) if (temp != "+9999" and re.match("[01459]", q)): print "%s\t%s" % (year, temp)mapper.py(last_key, max_val) = (None, 0)for line in sys.stdin: (key, val) = line.strip().split("\t") if last_key and last_key != key: print "%s\t%s" % (last_key, max_val) (last_key, max_val) = (key, int(val)) else: (last_key, max_val) = (key, max(max_val, int(val)))if last_key: print "%s\t%s" % (last_key, max_val)reduer.pycat dataFile | mapper.py | sort | reducer.pyHadoop Streaming](https://crownmelresort.com/image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-8-2048.jpg)
![def mapper(key,value): line = value.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if (temp != "+9999" and quality in "01459"): yield year, int(temp)def reducer(key,values): yield key,max(values)if __name__ == "__main__": import dumbodumbo.run(mapper,reducer,reducer)Dumbo](https://crownmelresort.com/image.slidesharecdn.com/pythonhadoopytrihug72010-100722182858-phpapp01/75/Megadata-With-Python-and-Hadoop-10-2048.jpg)
Megadata With Python and Hadoop
- 1. Processing MegadataWith Pythonand HadoopJuly 2010 TriHUGRyan Coxwww.asciiarmor.com
- 2.
- 3. high_temp=0forline inopen('1901'): line =line.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if(temp !="+9999"and quality in"01459"): high_temp=max(high_temp,float(temp)) printhigh_tempHow can we make this scale?( and do more interesting things )
- 5. JeffREY DEAN –Google - 2004“Our abstraction is in-spired by the map and reduce primitives present in Lisp and many other functional languages. We realized that most of our computations involved applying a map operation to each logical record in our input in order to compute a set of intermediate key/value pairs, and then applying a reduce operation to all the values that shared the same key, in order to combine the derived data appropriately. Our use of a functional model with user-specified map and reduce operations allows us to parallelize large computations easily and to use re-execution as the primary mechanism for fault tolerance.”
- 7. defmapper(line): line=line.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if(temp !="+9999"and quality in"01459"): returnfloat(temp) returnNoneoutput =map(mapper,open('1901')) printreduce(max,output)Mapreduce in pure python
- 8. for line insys.stdin:val = line.strip() (year, temp, q) = (val[15:19], val[87:92], val[92:93]) if (temp != "+9999" and re.match("[01459]", q)): print "%s\t%s" % (year, temp)mapper.py(last_key, max_val) = (None, 0)for line in sys.stdin: (key, val) = line.strip().split("\t") if last_key and last_key != key: print "%s\t%s" % (last_key, max_val) (last_key, max_val) = (key, int(val)) else: (last_key, max_val) = (key, max(max_val, int(val)))if last_key: print "%s\t%s" % (last_key, max_val)reduer.pycat dataFile | mapper.py | sort | reducer.pyHadoop Streaming
- 9.
- 10. def mapper(key,value): line = value.strip() (year, temp, quality) = (line[15:19], line[87:92], line[92:93]) if (temp != "+9999" and quality in "01459"): yield year, int(temp)def reducer(key,values): yield key,max(values)if __name__ == "__main__": import dumbodumbo.run(mapper,reducer,reducer)Dumbo
- 11. Ability to passaround Python objects
- 12. Job / IterationAbstraction
- 13. Counter / StatusAbstraction
- 14.
- 15. Ability to usenon-Java combiners
- 16. Built-in library ofmappers / reducers
- 17. Excellent way tomodel MR algorithmsDumbo
- 18. CLI – API– Web ConsoleAmazon Elastic MapReduce is a web service that enables businesses, researchers, data analysts, and developers to easily and cost-effectively process vast amounts of data. It utilizes a hosted Hadoop framework running on the web-scale infrastructure of Amazon Elastic Compute Cloud (Amazon EC2) and Amazon Simple Storage Service (Amazon S3).Elastic Map REduce
- 19.
- 20. Use Hadoop’s JobTracker or Amazon’s ‘Debugger’Elastic Map REduce
- 21.
- 22.
- 23. Quiz: How wouldyou do this?
- 24. Map Reduce AlgorithmsARE differentBFS(G, s) // G is the graph and s is the starting nodefor each vertex u ∈ V [G] - {s}do color[u] ← WHITE // color of vertex ud[u] ← ∞ // distance from source s to vertex uπ[u] ← NIL // predecessor of ucolor[s] ← GRAYd[s] ← 0π[s] ← NILQ ← Ø // Q is a FIFO - queueENQUEUE(Q, s) while Q ≠ Ø // iterates as long as there are gray vertices. do u ← DEQUEUE(Q)for each v ∈ Adj[u]do if color[v] = WHITE // discover the undiscovered adjacent verticesthen color[v] ← GRAY // enqueued whenever painted grayd[v] ← d[u] + 1π[v] ← uENQUEUE(Q, v)color[u] ← BLACK // painted black whenever dequeued
- 25. > m =function() { emit(this.user_id, 1); } > r = function(k,vals) { return 1; } > res = db.events.mapReduce(m, r, { query : {type:'sale'} }); > db[res.result].find().limit(2) { "_id" : 8321073716060 , "value" : 1 } { "_id" : 7921232311289 , "value" : 1 }MongoDB> {ok, [R]} = Client:mapred([{<<"groceries">>, <<"mine">>}, {<<"groceries">>, <<"yours">>}], [{'map', {'qfun', Count}, 'none', false}, {'reduce', {'qfun', Merge}, 'none', true}]).RiakMap Reduce Elsewhere
- 26. Definitive Guide Hadoophttp://www.hadoopbook.comDumbohttp://dumbotics.com/http://github.com/klbostee/dumbo/ElasticMap Reducehttp://aws.amazon.com/Botohttp://github.com/botoGetting Started Slideshttp://www.slideshare.net/pacoid/getting-started-on-hadoopLearn MOre
- 27.
Editor's Notes
- #2 This template can be used as a starter file for a photo album.