Downloaded 96 times
![RDDs
Think of an RDD[T] as an immutable, distributed collection
of objects of type T
• Resilient => Can be reconstructed in case of failure
• Distributed => Transformations are parallelizable
operations
• Dataset => Data loaded and partitioned across cluster
nodes (executors)](https://image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-26-2048.jpg)
![RDD[T]
Data distribution hierarchy:
- RDD[T]
- Elements
[ x1, x2 ]
[ x10 ]
[ x8,x5,x6 ]
[ x11 ]
[ x14,x13 ]
[ x9,x16 ]
[ x3 ]
[ x7,x12 ]
[ x15 ]
[ x17,x4 ]
Executor 1
- Executors
- Partitions
Executor 2 Executor 3 Executor 4](https://image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-27-2048.jpg)
![Spark Streaming
When you have big fat streams behaving as one single
collection
t
DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
DStreams: Discretized Streams (= Sequence of RDDs)](https://image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-31-2048.jpg)
![Mashup
prediction
Sample [NA20332] is in cluster #0 for population Some( ASW)
Sample [NA20334] is in cluster # 2 for population Some( ASW)
Sample [HG00120] is in cluster # 2 for population Some( GBR)
Sample [NA18560] is in cluster # 1 for population Some( CHB)](https://image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-41-2048.jpg)
![RDDs
Think of an RDD[T] as an immutable, distributed collection
of objects of type T
• Resilient => Can be reconstructed in case of failure
• Distributed => Transformations are parallelizable
operations
• Dataset => Data loaded and partitioned across cluster
nodes (executors)](https://crownmelresort.com/image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-26-2048.jpg)
![RDD[T]
Data distribution hierarchy:
- RDD[T]
- Elements
[ x1, x2 ]
[ x10 ]
[ x8,x5,x6 ]
[ x11 ]
[ x14,x13 ]
[ x9,x16 ]
[ x3 ]
[ x7,x12 ]
[ x15 ]
[ x17,x4 ]
Executor 1
- Executors
- Partitions
Executor 2 Executor 3 Executor 4](https://crownmelresort.com/image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-27-2048.jpg)
![Spark Streaming
When you have big fat streams behaving as one single
collection
t
DStream[T]
RDD[T] RDD[T] RDD[T] RDD[T] RDD[T]
DStreams: Discretized Streams (= Sequence of RDDs)](https://crownmelresort.com/image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-31-2048.jpg)
![Mashup
prediction
Sample [NA20332] is in cluster #0 for population Some( ASW)
Sample [NA20334] is in cluster # 2 for population Some( ASW)
Sample [HG00120] is in cluster # 2 for population Some( GBR)
Sample [NA18560] is in cluster # 1 for population Some( CHB)](https://crownmelresort.com/image.slidesharecdn.com/geeksa-150130082242-conversion-gate02/75/What-is-Distributed-Computing-Why-we-use-Apache-Spark-41-2048.jpg)
Uploaded byAndy Petrella
What is Distributed Computing, Why we use Apache Spark
The document discusses the advancements in big data technologies, particularly focusing on Apache Spark and its advantages over Hadoop. It highlights the importance of distributed environments and how Spark's features, such as resilient distributed datasets (RDDs) and in-memory caching, enable faster and smarter data processing for various use cases, including machine learning and genomics. The authors emphasize Spark's efficiency in handling large datasets and its capabilities in real-time processing and complex model training.
More Related Content
Similar to What is Distributed Computing, Why we use Apache Spark
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
What is Distributed Computing, Why we use Apache Spark
- 1. BigData, newborn technologies evolvingfast. Why Apache Spark outruns Apache Hadoop Andy Petrella, Nextlab Xavier Tordoir, SilicoCloud
- 2. Andy @Noootsab, I am @NextLab_beowner @SparkNotebook creator @Wajug co-driver @Devoxx4Kids organizer Maths & CS Data lover: geo, open, massive Fool Who are we? Xavier @xtordoir SilicoCloud -> Physics -> Data analysis -> genomics -> scalable systems -> ...
- 3. So what... Part I ●What ○ distributed resources ○ data ○ managers ● Why: ○ fastest ○ smartest ○ biggest ● How: ○ Map Reduce ○ Limitations ○ Extensions PART II ● Spark ○ Model ○ Caching and lineage ○ Master and Workers ○ Core example ● Beyond Processing ○ Streaming ○ SQL ○ GraphX ○ MLlib ○ Example ● Use cases ○ Parallel batch processing of timeseries ○ ADAM
- 4. Part I: TheDistributed Age
- 5. What is adistributed environment Computations needs three kind of resources: ● CPU ● MEM ● Data storage However, it’s hard to extent each of them at will on a single machine
- 6. What is adistributed environment Lacking of one of these will result in higher response time or reduced accuracy. Unfortunately, it doesn’t matter how parallelized is the algorithm or optimized are the computations If the solution can’t be inside, it must be outside.
- 7. What is adistributed environment
- 8. Distributed File System Youhave 100 nodes in your cluster, but only 1 dataset. Will you replicate it on all nodes? Extended case: your dataset is 1 Zettabyte (10⁹Tb)? Lonesome solution: ● split the file on nodes ● axing the algorithm to access local data subsets
- 9. HDFS towards Tachyon HadoopDistributed File System Implements GoogleFS Store and read files splitted and replicated on nodes 1Zb file = 8E12 x 128Mb files IOPs are expensive and require more CPU clocks than DRAM access Hence... Tachyon: memory-centric distributed file system
- 10. Nodes will fail,jobs cannot We need resilience Management Resources are generally fewer than required by algorithm. We need scheduling The requirements are fluctuating We need elasticity
- 11. Mesos and Marathon Mesos:High available cluster manager Nodes: attach or remove them on the fly Nodes are offering resources -- Applications accept them Node crash: the application restarts the assigned tasks Marathon: Meta application on Mesos Application crash: automatically restarted on different node
- 12. Why: for everybodyand now ? Fastest: 1. Time to result 2. Near real time processing
- 13. Runtime is smaller,Dev lifecyle is shorter → no synchronization-hell It can even be really interactive → consoles or notebooks tools. Why for everybody and now
- 14. Why for everybodyand now No bottlenecks → new-coming data are readily available for processing Opens the doors for online models!
- 15. Why for everybodyand now Smartest: train more and more models, ensembling lots of them is no more a problem More complex modelling can be tackled if required
- 16. Why for everybodyand now Accessing an higher level of accuracy is tricky and might require lots and lots of models. Running a model takes quite some time, specially if the data has to be read every single time. Example: Netflix contest winner (AT&T labs) ensembled 500 models to gain 10% accuracy. Although in 2009 it wasn’t possible to use it in production, today this could change.
- 17. Why for everybodyand now Biggest: no need for sampling big datasets … … That’s it!
- 18. How!? Google papers stimulatedthe open software community, hence competitive tools now exist. In the area of computation in distributed environment, there are two disruptive papers: ● Google’s Mapreduce ● Berkeley’s Spark
- 19. How!? MapReduce (Google whitepaper 2004): Programming model for distributed data intensive computations Helps dealing with parallelization, fault-tolerance, data distribution, load balancing
- 20. Functions: Map ≅ transformdata to key value pairs Reduce ≅ aggregate key value pairs per key (e.g. sum, max, count) Mappers and Reducers are sent to data location (nodes) How!?
- 21. Map Reduce: apply abinary associative operator on all elements Image from RxJava: https://github.com/ReactiveX/RxJava/wiki/Transforming-Observables How!?
- 22. Hadoop implementation hassome limitations Mappers and Reducers ship functions to data while java is not a functional language ⇒ Composability is difficult and more IO/network operations are required Iterative algorithms (e.g. stochastic gradient) have to read data at each step (while data has not changed, only parameters) How!?
- 23. How!? MapReduce on steroids I)Functional paradigm: - process built lazily based on simple concepts - Map and Reduce are two of them II) Cache data in memory. No more IO.
- 24. So what... Part I ●What ○ distributed resources ○ data ○ managers ● Why: ○ fastest ○ smartest ○ biggest ● How: ○ Map Reduce ○ Limitations ○ Extensions PART II ● Spark ○ Model ○ Caching and lineage ○ Master and Workers ○ Core example ● Beyond Processing ○ Streaming ○ SQL ○ GraphX ○ MLlib ○ Example (notebook) ● Use cases ○ Parallel batch processing of timeseries ○ ADAM
- 25. Part II: Sparkto the Rescue
- 26. RDDs Think of anRDD[T] as an immutable, distributed collection of objects of type T • Resilient => Can be reconstructed in case of failure • Distributed => Transformations are parallelizable operations • Dataset => Data loaded and partitioned across cluster nodes (executors)
- 27. RDD[T] Data distribution hierarchy: -RDD[T] - Elements [ x1, x2 ] [ x10 ] [ x8,x5,x6 ] [ x11 ] [ x14,x13 ] [ x9,x16 ] [ x3 ] [ x7,x12 ] [ x15 ] [ x17,x4 ] Executor 1 - Executors - Partitions Executor 2 Executor 3 Executor 4
- 28. Execution Execution is splitin fundamental units: Tasks Tasks running in parallel are grouped in Stages
- 29.
- 30.
- 31. Spark Streaming When youhave big fat streams behaving as one single collection t DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] DStreams: Discretized Streams (= Sequence of RDDs)
- 32. Spark SQL Mapping: RDD-> “table”, Element Field -> “column”
- 33. MLLib: Distributed ML Classification ●linear SVM, logistic regression, classification trees, naive Bayes Models Regression ● SVM, regression trees, linear regression (regularized) Clustering & dimensionality reduction ● singular value decomposition, PCA, k-means clustering “The library to teach them all”
- 34. GraphX Connecting the dots Graphprocessing at scale. > Take edges > Link nodes > Combine/Send messages
- 35. Use cases examples -Parallel batch processing of time series - Bayesian Network in financial market - IoT platform (Lambda architecture) - OpenStreetMap cities topologies classification - Markov Chain in Land Use/Land Cover prediction - Genomics: ADAM
- 36. Genomics Biological systems arevery complex One human sequence is 60Gb
- 37. ADAM Credits: AmpLab (UCBerkeley)
- 38. Stratification using 1000Genomes http://www.1000genomes.org/ ref:http://upload.wikimedia.org/wikipedia/en/e/eb/Genetic_Variation.jpg
- 39. Machine Learning model Clustering:KMeans ref: http://en.wikipedia.org/wiki/K-means_clustering
- 40. Machine Learning model MLLib,KMeans MLLib: ● Machine Learning Algorithms ● Data structures (e.g. Vector)
- 41. Mashup prediction Sample [NA20332] isin cluster #0 for population Some( ASW) Sample [NA20334] is in cluster # 2 for population Some( ASW) Sample [HG00120] is in cluster # 2 for population Some( GBR) Sample [NA18560] is in cluster # 1 for population Some( CHB)
- 42. Mashup #0 #1 #2 GBR0 0 89 ASW 54 0 7 CHB 0 97 0
- 43.
- 44. Eggo project (publicgenomics data in ADAM format on s3) We… 1000genomes in ADAM format on S3. Open Source GA4GH Interop services implementation Machine learning on 1000genomes Genomic data and distributed computing
- 45. The end (ofthe slides) Thanks for your attention! Xavier Tordoir xavier@silicocloud.eu Andy Petrella andy.petrella@nextlab.be