Download as PDF, PPTX
![Info
Analysis
Filter
[ json ]
Neural Network
How can it be addressed?](https://image.slidesharecdn.com/imcs20152devimplementingahighlyscalablein-memorystockprediction-150716213744-lva1-app6892/75/IMCSummit-2015-Day-2-Developer-Track-Implementing-a-Highly-Scalable-In-Memory-Stock-Prediction-System-with-Apache-Geode-R-and-Spring-XD-11-2048.jpg)
![Info
Analysis
Filter
[ json ]
Neural Network
How can it be addressed?](https://crownmelresort.com/image.slidesharecdn.com/imcs20152devimplementingahighlyscalablein-memorystockprediction-150716213744-lva1-app6892/75/IMCSummit-2015-Day-2-Developer-Track-Implementing-a-Highly-Scalable-In-Memory-Stock-Prediction-System-with-Apache-Geode-R-and-Spring-XD-11-2048.jpg)
Uploaded byIn-Memory Computing Summit
IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
The document discusses a stock prediction system utilizing open-source software, focusing on machine learning techniques like neural networks and real-time data processing. It highlights challenges in traditional models and emphasizes the need for real-time, stream-based analytics. The architecture proposed includes the integration of various data sources with a framework for scalable and efficient processing using tools like Spring XD and GemFire.
More Related Content
Similar to IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
More from In-Memory Computing Summit
IMCSummit 2015 - Day 2 Developer Track - Implementing a Highly Scalable In-Memory Stock Prediction System with Apache Geode, R and Spring XD
- 1. © Copyright 2014Pivotal. All rights reserved.© Copyright 2014 Pivotal. All rights reserved. A Stock Prediction System using open-source software Fred Melo fmelo@pivotal.io @fredmelo_br 1 William Markito wmarkito@pivotal.io @william_markito
- 3. © Copyright 2014Pivotal. All rights reserved. It's all about DATA Data Sources Look for patterns Prediction
- 4. © Copyright 2014Pivotal. All rights reserved.
- 5. © Copyright 2014Pivotal. All rights reserved. 5 Machine Learning is the answer Neural Networks Clustering Genetic Algorithms
- 6. © Copyright 2014Pivotal. All rights reserved. Train with historical dataset Apply model to the new input Applying Machine Learning
- 7. Hard to addnew data sources Why? Hard to scale Why so hard? Hard to make it real-time
- 8. Traditional models arereactive and static HDFS Data Lake Store Analytics Hard to change Labor intensive Inefficient No real-time information ETL based Data-source specific
- 9. Stream-based, real-time closed-loopanalytics are needed HDFSData Lake Expert System / Machine Learning In-Memory Real- Time Data Continuous Learning Continuous Improvement Continuous Adapting Data Stream Pipeline Multiple Data Sources Real-Time Processing Store Everything
- 10. Info Analysis Look at pasttrends (for similar input) Evaluate current input Score / Predict Neural Network How can it be addressed?
- 11. Info Analysis Filter [ json ] NeuralNetwork How can it be addressed?
- 12. Info Analysis Filter Enrich Neural Network Howcan it be addressed?
- 13. Info Analysis Neural Network Filter EnrichTransform How can it be addressed?
- 14. Info Analysis Filter Enrich Transform NeuralNetwork How can it be addressed?
- 15. Info Analysis Filter Enrich Transform Transform NeuralNetwork How can it be addressed?
- 16. Neural Network In-Memory DataGrid Real-time scoring How can it be addressed? Train
- 17. Neural Network In-Memory DataGrid Front-end Update Push How can it be addressed?
- 18. Ingest Transform Sink SpringXD Store/ Analyze Fast Data Distributed Computing Predict / Machine Learning Other Sources and Destinations JMS Streaming real-time analytics architecture
- 19. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 20. SpringXD INGEST / SINKPROCESS ANALYZE • Little or no coding required • Dozens of built-in connectors • Seamless integration with Kafka, Sqoop • Create new connectors easily using Spring • Call Spark, Reactor or RxJava • Built-in configurable filtering, splitting and transformation • Out-of-box configurable jobs for batch processing • Import and invoke PMML jobs easily • Call Python, R, Madlib and other tools • Built-in configurable counters and gauges Data Stream Pipelining
- 21. SpringXD XD NodesXD NodesXDNodesXD Nodes Ingest SpringXD Split Filter Transform Sink XD admin XD Nodes Ingest Split Filter Transform Sink Stream Deployment Messaging Scale-Out and HA Architecture
- 22. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 23.
- 24. Partitioned Regions GemFire Server1GemFire Server2 Primary'to'redundant'replica1on' Primary 0 2 4 6 Redundant 1 3 5 7 Region A Region B Region A Region B Primary 1 3 5 7 Redundant 0 2 4 6
- 25. Event handling GemFire'Server' Region'A' ' '''''''''' subscrip4on Region'A' pool6name=ServerPool' ' ''''''''' X GemFire'Client'1' pool'"ServerPool"' (with'or'without'' subscrip4ons'enabled) Region'A' pool6name=ServerPool' ' ''''''''' X GemFire'Client'2' pool'"ServerPool"' (with'subscrip4ons'enabled,' 'interest'register'in'X,'receiveValues=true) X Distributed'System Update'/'Create 1 2 3 3 4 X Thepool propagates the event to the cache server, where the region is updated. The server distributes the event to its peers and also places it into the subscription queue for Client 2.
- 26. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 27.
- 28.
- 29. medium avg (x+1) relative strength (x) mediumavg (x) price(x) Neural Network
- 30.
- 31. Transform Sink SpringXD Extensible Open-Source Fault-Tolerant Horizontally Scalable Cloud-Native HTTP MachineLearning Fast Data Filter Predict Sink HTTP Split Dashboard Push Demo Architecture
- 33.
- 34.
- 35.
- 37. A NEW PLATFORMFOR A NEW ERA