Download as PDF, PPTX
Uploaded byViktor Gamov
[OracleCode SF] In memory analytics with apache spark and hazelcast
The document discusses the integration of Apache Spark with Hazelcast's in-memory data grid, highlighting the speed advantages of Spark and its fault-tolerance with resilient distributed datasets (RDDs). It provides code examples for configuration and uses cases, emphasizing parallel processing and operational benefits. Additionally, it mentions limitations regarding data updates while reading from Spark, suggesting potential issues like cursor inaccuracies.
More Related Content
![[OracleCode - SF] Distributed caching for your next node.js project](https://cdn.slidesharecdn.com/ss_thumbnails/distributedcachingforyournextnode-170302181006-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Jfokus] Riding the Jet Streams](https://cdn.slidesharecdn.com/ss_thumbnails/ridingthejetstreams-jfokus-02-08-2017-170208120631-thumbnail.jpg?width=640&height=640&fit=bounds)
![[NYJavaSig] Riding the Distributed Streams - Feb 2nd, 2017](https://cdn.slidesharecdn.com/ss_thumbnails/ridingdistributedstreams-nyjavasig-02-02-2017-170203005616-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Codemash] Caching Made "Bootiful"!](https://cdn.slidesharecdn.com/ss_thumbnails/cachingmadebootifulcodemash11217-170112212321-thumbnail.jpg?width=640&height=640&fit=bounds)
![[JokerConf] Верхом на реактивных стримах, 10/13/2016](https://cdn.slidesharecdn.com/ss_thumbnails/ridingthejetstreams-jokerconf-10-13-2016-161014221528-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to [OracleCode SF] In memory analytics with apache spark and hazelcast
![[JBreak] Блеск И Нищета Распределенных Стримов - 04-04-2017](https://cdn.slidesharecdn.com/ss_thumbnails/gamov-jbreak-04-04-2017-170404164301-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DataSciCon] Divide, distribute and conquer stream v. batch](https://cdn.slidesharecdn.com/ss_thumbnails/dividedistributeandconquer-streamv-171201225205-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Philly ETE] Java Puzzlers NG](https://cdn.slidesharecdn.com/ss_thumbnails/javapuzzlerss02-phillyete-170418150118-thumbnail.jpg?width=640&height=640&fit=bounds)
![[Philly JUG] Divide, Distribute and Conquer: Stream v. Batch](https://cdn.slidesharecdn.com/ss_thumbnails/dividedistributeandconquer-streamv-170920171540-thumbnail.jpg?width=640&height=640&fit=bounds)
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
[OracleCode SF] In memory analytics with apache spark and hazelcast
- 1. @gamussa @hazelcast #oraclecode IN-MEMORYANALYTICS with APACHE SPARK and HAZELCAST
- 2. @gamussa @hazelcast #oraclecode SolutionsArchitect Developer Advocate @gamussa in internetz Please, follow me on Twitter I’m very interesting © Who am I?
- 3. @gamussa @hazelcast #oraclecode What’sApache Spark? Lightning-Fast Cluster Computing
- 4. @gamussa @hazelcast #oraclecode Runprograms up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.
- 5. @gamussa @hazelcast #oraclecode Whento use Spark? Data Science Tasks when questions are unknown Data Processing Tasks when you have to much data You’re tired of Hadoop
- 6. @gamussa @hazelcast #oraclecode SparkArchitecture
- 7.
- 8.
- 9. @gamussa @hazelcast #oraclecode ResilientDistributed Datasets (RDD) are the primary abstraction in Spark – a fault-tolerant collection of elements that can be operated on in parallel
- 10.
- 11. @gamussa @hazelcast #oraclecode RDDOperations
- 12. @gamussa @hazelcast #oraclecode operationson RDDs: transformations and actions
- 13. @gamussa @hazelcast #oraclecode transformationsare lazy (not computed immediately) the transformed RDD gets recomputed when an action is run on it (default)
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23. @gamussa @hazelcast #oraclecode parallelizedcollections take an existing Scala collection and run functions on it in parallel
- 24. @gamussa @hazelcast #oraclecode Hadoopdatasets run functions on each record of a file in Hadoop distributed file system or any other storage system supported by Hadoop
- 25. @gamussa @hazelcast #oraclecode What’sHazelcast IMDG? The Fastest In-memory Data Grid
- 26. @gamussa @hazelcast #oraclecode HazelcastIMDG is an operational, in-memory, distributed computing platform that manages data using in-memory storage, and performs parallel execution for breakthrough application speed and scale
- 27. @gamussa @hazelcast #oraclecode High-Density Caching In-Memory DataGrid Web Session Clustering Microservices Infrastructure
- 28. @gamussa @hazelcast #oraclecode What’sHazelcast IMDG? In-memory Data Grid Apache v2 Licensed Distributed Caches (IMap, JCache) Java Collections (IList, ISet, IQueue) Messaging (Topic, RingBuffer) Computation (ExecutorService, M-R)
- 29.
- 30.
- 31. @gamussa @hazelcast #oraclecode finalSparkConf sparkConf = new SparkConf() .set("hazelcast.server.addresses", "localhost") .set("hazelcast.server.groupName", "dev") .set("hazelcast.server.groupPass", "dev-pass") .set("hazelcast.spark.readBatchSize", "5000") .set("hazelcast.spark.writeBatchSize", "5000") .set("hazelcast.spark.valueBatchingEnabled", "true"); final JavaSparkContext jsc = new JavaSparkContext("spark://localhost:7077", "app", sparkConf); final HazelcastSparkContext hsc = new HazelcastSparkContext(jsc); final HazelcastJavaRDD<Object, Object> mapRdd = hsc.fromHazelcastMap("movie"); final HazelcastJavaRDD<Object, Object> cacheRdd = hsc.fromHazelcastCache("my- cache");
- 32. @gamussa @hazelcast #oraclecode finalSparkConf sparkConf = new SparkConf() .set("hazelcast.server.addresses", "localhost") .set("hazelcast.server.groupName", "dev") .set("hazelcast.server.groupPass", "dev-pass") .set("hazelcast.spark.readBatchSize", "5000") .set("hazelcast.spark.writeBatchSize", "5000") .set("hazelcast.spark.valueBatchingEnabled", "true"); final JavaSparkContext jsc = new JavaSparkContext("spark://localhost:7077", "app", sparkConf); final HazelcastSparkContext hsc = new HazelcastSparkContext(jsc); final HazelcastJavaRDD<Object, Object> mapRdd = hsc.fromHazelcastMap("movie"); final HazelcastJavaRDD<Object, Object> cacheRdd = hsc.fromHazelcastCache("my- cache");
- 33. @gamussa @hazelcast #oraclecode finalSparkConf sparkConf = new SparkConf() .set("hazelcast.server.addresses", "localhost") .set("hazelcast.server.groupName", "dev") .set("hazelcast.server.groupPass", "dev-pass") .set("hazelcast.spark.readBatchSize", "5000") .set("hazelcast.spark.writeBatchSize", "5000") .set("hazelcast.spark.valueBatchingEnabled", "true"); final JavaSparkContext jsc = new JavaSparkContext("spark://localhost:7077", "app", sparkConf); final HazelcastSparkContext hsc = new HazelcastSparkContext(jsc); final HazelcastJavaRDD<Object, Object> mapRdd = hsc.fromHazelcastMap("movie"); final HazelcastJavaRDD<Object, Object> cacheRdd = hsc.fromHazelcastCache("my- cache");
- 34. @gamussa @hazelcast #oraclecode finalSparkConf sparkConf = new SparkConf() .set("hazelcast.server.addresses", "localhost") .set("hazelcast.server.groupName", "dev") .set("hazelcast.server.groupPass", "dev-pass") .set("hazelcast.spark.readBatchSize", "5000") .set("hazelcast.spark.writeBatchSize", "5000") .set("hazelcast.spark.valueBatchingEnabled", "true"); final JavaSparkContext jsc = new JavaSparkContext("spark://localhost:7077", "app", sparkConf); final HazelcastSparkContext hsc = new HazelcastSparkContext(jsc); final HazelcastJavaRDD<Object, Object> mapRdd = hsc.fromHazelcastMap("movie"); final HazelcastJavaRDD<Object, Object> cacheRdd = hsc.fromHazelcastCache("my- cache");
- 35.
- 36.
- 37. @gamussa @hazelcast #oraclecode DATASHOULD NOT BE UPDATED WHILE READING FROM SPARK
- 38.
- 39. @gamussa @hazelcast #oraclecode MAPEXPANSION SHUFFLES THE DATA INSIDE THE BUCKET
- 40. @gamussa @hazelcast #oraclecode CURSORDOESN’T POINT TO CORRECT ENTRY ANYMORE, DUPLICATE OR MISSING ENTRIES COULD OCCUR
- 41.
- 42. @gamussa @hazelcast #oraclecode THANKS! Anyquestions? You can find me at @gamussa viktor@hazelcast.com