Download as PDF, PPTX
Uploaded byrhatr
Apache Spark: killer or savior of Apache Hadoop?
The document discusses Apache Spark's relationship with Hadoop, analyzing its functionalities and innovations, particularly how Spark simplifies data processing for data scientists. It highlights Spark's features, including resilient distributed datasets (RDDs), capabilities for stream processing, and machine learning integration, contrasting them with traditional MapReduce methods. Additionally, the document provides insights into the evolving ecosystem of big data technologies, emphasizing Spark's advantages and its emergence as a significant tool in modern data architectures.
More Related Content
Similar to Apache Spark: killer or savior of Apache Hadoop?
Apache Spark: killer or savior of Apache Hadoop?
- 1. Apache Spark: killer orsavior of Apache Hadoop? Roman Shaposhnik Director of Open Source @Pivotal (Twitter: @rhatr)
- 2. Who’s this guy? • Director of Open Source (building a team of OS contributors) • Apache Software Foundation guy (Member, VP of Apache Incubator, committer on Hadoop, Giraph, Sqoop, etc) • Used to be root@Cloudera • Used to be PHB@Yahoo! (original Hadoop team) • Used to be a hacker at Sun microsystems (Sun Studio compilers and tools)
- 3.
- 4.
- 5. 40 minute tofigure out Hadoop vs. Spark
- 6. 40 minute tofigure out Hadoop++ == Spark
- 7. 40 minute tofigure out Hadoop + Spark
- 8. 40 minute tofigure out
- 9. Long, long timeago… HDFS ASF Projects FLOSS Projects Pivotal Products MapReduce
- 10. In a blinkof an eye HDFS Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Crunch Mahout Spark Shark Streaming MLib GraphX Impala HAWQ SpringXD MADlib Hamster PivotalR YARN Tachyon
- 11. A Spark view? HDFS SqoopFlume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie Hadoop UI Hue SolrCloud Phoenix HBase Spark Shark Streaming MLib GraphX SpringXD YARN Tachyon
- 12.
- 13. Principle #1 HDFS isthe datalake
- 14.
- 15.
- 16. HDFS: decoupled storage … MR HDFS MR
- 18. Anatomy of MapReduce da c a b c a 3 b 1 c 2 a 1 b 1 c 1 a 1 c 1 a 1 a 1 1 1 b 1 c 1 1 HDFS mappers reducers HDFS
- 19. Principle #2 MR isassembly language
- 20.
- 21.
- 22.
- 23. Principle #3 MR: YARN+ library
- 24. What’s wrong withMR? Source: UC Berkeley Spark project (just the image)
- 25. Principle #4 $ grep–R | awk | sort …
- 26. Spark philosophy • Make lifeeasy for Data Scientists • Provide well documented and expressive APIs • Powerful Domain Specific Libraries • Easy integration with storage systems • Caching to avoid data movement • Well defined releases, stable API
- 27. Spark innovations • Resilient DistribtuedDatasets (RDDs) • Distributed on a cluster • Manipulated via parallel operators (map, etc.) • Automatically rebuilt on failure • A parallel ecosystem • A solution to iterative and multi-stage apps
- 28. RDDs warnings = textFile(…).filter(_.contains(“warning”)) .map(_.split(‘‘)(1)) HadoopRDD path = hdfs:// FilteredRDD contains… MappedRDD split…
- 29. Parallel operators • map, reduce • sample,filter • groupBy, reduceByKey • join, leftOuterJoin, rightOuterJoin • union, cross
- 30. How do Iuse it? val file = spark.textFile(hdfs://...) val counts = file.flatMap(line = line.split( )) .map(word = (word, 1)) .reduceByKey(_ + _) counts.saveAsTextFile(hdfs://...)
- 31. Principle #5 Memory isthe new disk
- 32. RDDs are thefoundation • SQL • Graph • ML • Streaming
- 33. Spark SQL • Lib inSpark Core that models RDDs as rels. • SchemaRDD • Replaces Shark • Lightweight with no code from Hive • Import/Export into different storage formats • Columnar storage (as in Shark)
- 34. Spark Streaming • Extend Sparkto do large scale stream processing • Simple, batch like API with RDDs • Single semantics for both real time and high latency
- 35.
- 36.
- 37. Spark GraphX • Pregel (BSP)(formerly know as Bagel) • Graph-centric modeling • Unification of processing • No more MR trickery
- 38. You killed ApacheGiraph?
- 39. MLbase • Machine Learning toolset • MatLabfor scale out computing • Built on Spark Mlib • Classification, Regression, Colab. Filtering, etc.
- 40. What is reallyhappening? HDFS Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Crunch Mahout Spark Shark Streaming MLib GraphX Impala HAWQ SpringXD MADlib Hamster PivotalR YARN Tachyon
- 41. Principle #6 Spark: theecosystem
- 42. May be itsnot so bad server 1 server N
- 43. But HDFS/YARN aresafe? HDFS, Ceph, S3, NAS, etc. New HDFS New YARN
- 44. What is *really*going on? • 2009 Research at UCB, written in Scala • 2010 Open Sourced • 2013 Accepted into Apache Incubator • 2013 Databricks formed ($14M funding) • 2014 Becomes TLP with ASF • 2014 Spark 1.0 is out • 2014 Databricks gets an extra $33M
- 45. Bigdata: brought toU by ASF • 50% ML traffic • 100-200 contributors across 25-35 companies • More active than Hadoop • Cross-pollination with other TLPs
- 46. Principle #7 Where Hadoopwas ‘09
- 47. This is howhardening looks
- 48. What is Hadoop? Hadoop!= MR + HDFS
- 49. The ecosystem • Apache HBase • ApacheCrunch, Pig, Hive and Phoenix • Apache Giraph • Apache Oozie • Apache Mahout • Apache Sqoop and Flume
- 50. Principle #8 Spark: analternative backend
- 51. Spark is bestfor cloud
- 52. Principle #9 Memory isexpensive
- 53. What’s new? • True elasticity • Resourcepartitioning • Security • Data marketplace • Multi datacenter deployments
- 54. Hadoop Maturity ETL Offload Accommodatemassive data growth with existing EDW investments Data Lakes Unify Unstructured and Structured Data Access Big Data Apps Build analytic-led applications impacting top line revenue Data-Driven Enterprise App Dev and Operational Management on HDFS Data Architecture
- 55. Pivotal HD onPivotal CF Ÿ Enterprise PaaS Management System Ÿ Flexible multi-language ‘buildpack’ architecture Ÿ Deployed applications enjoy built-in services Ÿ On-Premise Hadoop as a Service Ÿ Single cluster deployment of Pivotal HD Ÿ Developers instantly bind to shared Hadoop Clusters Ÿ Speeds up time-to-value
- 56. Pivotal’s view Data SciencePlatform Tachyon/Gem Cluster Manager MR Application Stream Server MPP SQL Data Lake / HDFS / Virtual Storage GemFireXD ...ETC Hadoop HDFS Isilon App Dev / Ops MLbase Streaming Legacy Systems Legacy Data Scientists Data Sources End Users SparkSQL
- 57. Principle #10 The rumorsof my death…
- 58. It will becalled Hadoop HDFS Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire with Tachyon Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Crunch Mahout Spark Shark Streaming MLib GraphX Impala HAWQ SpringXD MADlib Hamster PivotalR YARN
- 59. Spark recap • Is it“Big Data” (Yes) • Is it “Hadoop” (No) • It’s one of those “in memory” things, right (Yes) • JVM, Java, Scala (All) • Is it Real or just another shiny technology with a long, but ultimately small tail (Yes and ?)
- 60. A NEW PLATFORMFOR A NEW ERA
- 61. Credits • Wikipedia and Dilbert.com • ApacheSoftware Foundation • Scott Deeg • Milind Bhandarkar • Susheel Kaushik • Mak Gokhale
- 62.