Download as PDF, PPTX
Uploaded byJeffrey T. Pollock
Oracle Stream Analytics - Developer Introduction
The document provides an overview of Oracle Stream Analytics, outlining its capabilities in building real-time data processing pipelines and showcasing its features such as ETL, machine learning, and geospatial analytics. It emphasizes the integration with Goldengate, various industry use cases, and the potential applications in real-time decision-making across multiple sectors. Additionally, it serves as a forward-looking statement about Oracle's product directions with risks and disclaimers regarding future developments.
In this document
Powered by AI
Introduction by Jeff Pollock; forward-looking statements about Oracle products and development.
Streaming platforms enable real-time data processing; outline of use cases and key features.
Examples of streaming use cases across IT, finance, healthcare, and more; emphasizes real-time insights.
Key feature areas of GoldenGate Stream Analytics and its integration with data pipelines.
Roles of Data Engineers and Analysts in creating and visualizing data streams using user-friendly tools.
Integration of ML for real-time scoring, predictive analysis, and geo-spatial capabilities.
Methods for processing live data, transforming events, and monitoring database transactions.
Horizontal scalability, high availability, and architecture details for efficient data processing.
Sequential steps to build a streaming app, including data ingestion, patterns, and ML scoring.
Industry use cases, GoldenGate capabilities, and various platforms for effective data integration.
Overview of integration possibilities with cloud services, logical data zones, and ETL capabilities.
More Related Content
Similar to Oracle Stream Analytics - Developer Introduction
Oracle Stream Analytics - Developer Introduction
- 1. 1 Oracle Stream Analytics Overview Speakers JeffPollock, VP Product Development Copyright © 2019 Oracle and/or its affiliates.
- 2. The following isintended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
- 3. Industry and UseCases Oracle Stream Analytics Key Features Developer Experience Streaming Data Pipelines (ETL) with GoldenGate …for More Information Copyright © 2019 Oracle and/or its affiliates.
- 4. What is aStreaming Platform? Copyright © 2019 Oracle and/or its affiliates. A Streaming Platform is to build real-time data processing pipelines in a rapid fashion, at both small and very large scale, in order to provide actionable business insights and very fast data processing services
- 5. Key Streaming UseCases Copyright © 2019 Oracle and/or its affiliates. IT/Data Management Financial Services Transportation Telecom • Data Lake / Warehouse Ingest • Enterprise Data Services (REST, Pub/Sub) • Streaming ETL, Data Pipelines • DataOps (insights on operational data) • Fraud Detection • Risk Management • Real-time analysis of currency exchange data or commodities • Customer retention, realtime issue intervention • Tracking Containers, Delivery Vehicles, and other Assets • Vehicle Management • Passenger Alerts • Logistics and Route Optimization • Wifi Off-Loading • Video Analytics • Network Management • Security Operations • Geolocation Marketing • Mobile Data Processing Retail Manufacturing Utilities, Oil & Gas Healthcare • Real-time Personalized Offers • Markdown optimization • Dynamic pricing and forecasting • Shopping cart defections • Better store and shelf management • Customer retention, realtime issue intervention • Smart Inventory • Quality Control • Building Management • Logistics and Route Optimization • Outage Intelligence • Workforce Management • Real-time Drilling Analysis • Telemetry on critical assets • Medical Device Monitoring • In-home Patient Monitoring • Medical Fraud Detection • Safer Cities
- 6. For Building DataPipelines Copyright © 2019 Oracle and/or its affiliates. Ingest Database Events Select Processing Patterns Build Event Pipelines Serve Data Downstream Any GoldenGate event is included free, Kafka native events require full-use license Rich set of pre-built patterns can dramatically improve developer efficiency and time-to-value Tool can easily leverage geo-fencing, machine-learning, and other lookup data within the data stream Data can be delivered out to kafka, databases, or easily staged for downstream ETL jobs connect Messages, Events & Alerts Analytics Data {rest} Data Lake
- 7. MPP Data Processing Raw Data AComplete Solution Copyright © 2019 Oracle and/or its affiliates. GG Database Replication Open Source Stream Analytics GG for Big Data ETL Filter Aggregate Transform Correlate/Enrich Geo-fence Thresholds Business Rules Data Policies Queries Time Series Spatial Analytics Data Patterns Anomalies Classification Clustering Statistical Inference Regression Models Messages, Events & Alerts Analytics Data {rest} Data Lake LOW CODE HIGH SPEED LOW LATENCY DATAOPS
- 8. Part of theGoldenGate Platform Copyright © 2019 Oracle and/or its affiliates. Replication of Real-time Data Transactions & Events Stream Analytics ETL &ML DBMS Cloud Big Data NoSQL Streams Object Storage
- 9. Industry and UseCases Oracle Stream Analytics Key Features Developer Experience Streaming Data Pipelines (ETL) with GoldenGate …for More Information Copyright © 2019 Oracle and/or its affiliates.
- 10. 10 GG Stream Analytics– Key Feature Areas Interactive Designer UI Rich Set of Streaming Patterns Predictive Analysis and Machine Learning Location and Geospatial Analysis Integrated CDC with Oracle GoldenGate Robustness, Speed, and Scalability
- 11. Two User Personas Copyright© 2019 Oracle and/or its affiliates. Data Engineer • Creates custom queries and patterns • Defines complex windowing correlations • Installs and maintains Big Data and Messaging Environment • Manages lookup data sources • Tunes production pipelines for performance and high availability Data Analyst • Explores incoming data content • Defines visualizations and dashboards • Creates and maintains data rules • Chooses and applies patterns • Defines geo fences and spatial rules
- 12. Interactive Browser-based Designer Copyright© 2019 Oracle and/or its affiliates. Accessible to Non-Technical Users • Empower data analysts to enhance data with no coding skills required • Intuitive, always-on data view shows results of transformations as they are defined • Filter and correlate streams, apply rules, aggregate, calculate fields etc. Function extensibility via Java • Allow data engineers to provide custom stages and functions to be used by all team members Integrated Visualizations • Explore your business data live through various tables, charts and geospatial maps
- 13. Predictive Analysis andMachine Learning Copyright © 2019 Oracle and/or its affiliates. Real-time Scoring and Decision Making • Use Machine Learning models to make business decisions in real-time • Predict future outcomes such as equipment failures, customer behavior, fraud and security breaches • Re-import refined models for improved predictions Put Data Science in Production • Import Predictive Models created by data scientists and engineers in their own environment. • Import of PMML models for a variety of algorithms such as vector machines, association rules, Naive Bayes classifier, clustering models, text models, decision trees, and different regression models. • Hide model complexity for use by data analysts • Custom stages for access to external scoring systems Oracle R Enterprise Notebooks (Jupyter, Zeppelin, etc) Data Scientist Data Analyst/ Data Engineer
- 14. Location and Geo-SpatialCapabilities Copyright © 2019 Oracle and/or its affiliates. Interactive Spatial Design and Visualization • Show live location data on maps as events are processed • Track individual objects and highlight them based on different conditions, e.g. Red for violation Rich Geospatial Pattern Set • Correlate multiple objects through their spatial interaction • Detect speed, and proximity • Obtain address and city information from location and vice versa through Geocoding Scalable Definition of Areas and Geo-Fences • Define polygons through drawing borders on a map • Manage large amounts of shapes through spatial types in Oracle database.
- 15. GoldenGate Integrations Copyright ©2019 Oracle and/or its affiliates. Process and Analyze Live Data • Gain insights into business by analyzing live transactions – or schema events (DDL etc) • Transform and aggregate events to store into data lake in real time using filters, joins, rules, aggregations, splits, unions and other common operations. • Natively process GoldenGate change records and keep live aggregates based on change records Monitor your Database Transactions • Analyze statistics of ongoing database activity • Identify hot records with many changes, monitor sensitive tables or records for activity and exceeding thresholds. • Correlate different transactions, for example confirm that a request is acknowledged. • Identify unusual or fraudulent activity, such as records that are created and soon after deleted Real-time BI Big Data Lakes Business Process Operational Dashboards OLTP Database GoldenGate Kafka - Oracle Enterprise Hub Oracle Stream Analytics How does my data do right now? Alert and Act on critical issues OLTP Database GoldenGate Kafka - Oracle Enterprise Hub Oracle Stream Analytics Target Database
- 16. Rich Set ofStreaming Patterns Copyright © 2019 Oracle and/or its affiliates. Simplify Access to Complex Algorithms • Easy-to-use modules with user assistance in the designer • Pre-defined visualizations to provide immediate feedback • Accessible to data analysts Comprehensive Library of Patterns • Covers diverse areas such as anomaly detection, stream correlation, trend analysis, spatial functions • Duplicate, out-of-order, and missing event detection • Functions for financial, statistic, and log analytic operations
- 17. Robustness, Speed andScalability Copyright © 2019 Oracle and/or its affiliates. Horizontal Scalability through Spark clusters • High throughput by using highly parallelized in-memory processing by Spark • Efficient event correlations using Oracle’s CQL engine • Cloud-based elastic Spark clusters through Oracle Big Data Cloud • Scale clusters dynamically by adding more nodes • Distribute multiple web server nodes through load balancing High Availability through Cluster Redundancy • Exactly once query semantics • Recover failing worker nodes in Spark cluster without Data Loss
- 18. Architecture Copyright © 2019Oracle and/or its affiliates. LBR OSA Web-tier jetty OSA Web-tier jetty Metadata Store Web-Tier Cluster MPP Processing Pipeline Deployment & Data Access Data Streams From Applications Results, Alerts, Notifications • Pipeline Design • Interactive Analytics • Dashboards runtimedesign time
- 19. Industry and UseCases Oracle Stream Analytics Key Features Developer Experience Streaming Data Pipelines (ETL) with GoldenGate …for More Information Copyright © 2019 Oracle and/or its affiliates.
- 20. Oracle Stream Analytics 20Confidential– Oracle Internal/Restricted/Highly Restricted Data Pipeline GoldenGate Feeds Sensor Data Social Media Click Stream Geo Location Filter Aggregate Transform Correlate/Enrich Geo-fence Queries Time Windows Data Patterns Spatial Analytics Anomalies Classification Clustering Statistical Inference Regression Models Business Rules Policies Conditional Logic Notify/Publish Invoke/Execute Visualize Persist Data Ingestion Pre-processing Analysis Prediction Decisions Actions Ingest Transform and Correlate Act and Deliver
- 21. End-to-End Steps tobuild a Stream Application: 1. Create Connections, Stream, and References for Sources 21Confidential – Oracle Internal/Restricted/Highly Restricted Kafka, JMS, File, Database, or REST Connection Types Message Shape is detected from Kafka Topic
- 22. 2. Create GeographicalAreas / Geo Fences 22Confidential – Oracle Internal/Restricted/Highly Restricted Build Geo Fences manually or from DB repository
- 23. 3. Import PredictivePMML Model 23Confidential – Oracle Internal/Restricted/Highly Restricted Train and export PMML models from common ML tools such as R, SAS, H2O, etc.
- 24. 4. Create NewPipeline 24Confidential – Oracle Internal/Restricted/Highly Restricted Incoming messages are displayed automatically New Pipeline is immediately valid and active
- 25. 5. Add Joinsto Pipeline 25Confidential – Oracle Internal/Restricted/Highly Restricted Join Stream or Batch source Joined events are shown with color-coded fields
- 26. 6. Add Patternsto Pipeline 26Confidential – Oracle Internal/Restricted/Highly Restricted Choose from a library of vertical patterns Event locations are shown on map in real-time
- 27. 7. Add MLScoring to Pipeline 27Confidential – Oracle Internal/Restricted/Highly Restricted Refer to uploaded PMML model Map event fields into PMML model properties
- 28. 8. Add Targetto Pipeline 28Confidential – Oracle Internal/Restricted/Highly Restricted Send events to Kafka, JMS, or REST targets
- 29. 9. Publish Pipelineto Production 29Confidential – Oracle Internal/Restricted/Highly Restricted One-click deploy into production Spark cluster
- 30. Industry and UseCases Oracle Stream Analytics Key Features Developer Experience Streaming Data Pipelines (ETL) with GoldenGate …for More Information Copyright © 2019 Oracle and/or its affiliates.
- 31. More than 20Years of Innovation Copyright © 2019 Oracle and/or its affiliates. 1000’S OF CUSTOMERS GLOBALLY 1990’s – Database HA/DR 2000’s – OLTP Replication 2010 – Data Warehouse 2015 – Data Lake & Cloud KEY USE CASES & GROWTH PHASES: http://www.oracle.com/us/products/middleware/data- integration/oracle-goldengate-innovations-wp-5093027.pdf
- 32. GoldenGate Platform Capabilities Copyright© 2019 Oracle and/or its affiliates. Data Replication Data Lake Pipelines Stream Analytics Data High Availability • Oracle/Non-Oracle DB • Low Downtime Migrations Transaction Replication • OLTP/Reference Data Data Warehouse Loading • Non-invasive Capture • Realtime Staging Data Lake Ingest • High Fidelity Change Stream • Event-driven Realtime Pre-Processing • Filter, Correlate, Enrich Data Transformations • Streaming ETL Ops: Query, Aggregate, Lookup, etc. Data Operations (DataOps) • Low code development, iterate and work with production data Advanced Analytics • Time Series Analysis, Machine Learning, Geo-Spatial Dashboards • Active graphs and charts
- 33. For Oracle Cloud Copyright© 2019 Oracle and/or its affiliates. Virtual Machine Database Cloud Service Bare Metal Database Cloud Exadata Cloud Service Autonomous DB OCI Object Storage OCI Streaming Service Replication of Real-time Data Transactions & Events
- 34. For the Enterprise Copyright© 2019 Oracle and/or its affiliates. Replication of Real-time Data Transactions & Events GoldenGate Stream Analytics ETL &ML DBMS Cloud Big Data NoSQL Streams Object Storage
- 35. GoldenGate Marketplace –3 Steps to Productivity Step 1: Search OCI Marketplace Step 3: Run GG Web AppsStep 2: Provision GoldenGate on OCI Navigate to https://cloudmarketplace.oracle. com/marketplace/en_US/listing/ 58489224 Choose “Launch App” and click through ~4 pages of forms (existing OCI account is required) Access GoldenGate Microservices (Web Apps) from public IP Address given via OCI confirmation page FREE Promo Now! Copyright © 2019 Oracle and/or its affiliates.
- 36. Stream Analytics Marketplace Copyright© 2019 Oracle and/or its affiliates. OCI Marketplace GG Database Replication Open Source Stream Analytics GG for Big Data OCI Compute (any shape) OCI Block Store Trail File Kafka Topics MySQL Store Messages, Events & Alerts Analytics Data {rest} Data Lake LOW LATENCY HIGH SPEED LOW CODE DATAOPS
- 37. GoldenGate Integrations Copyright ©2019 Oracle and/or its affiliates. Process and Analyze Live Data • Gain insights into business by analyzing live transactions • Transform and aggregate events to store into data lake in real time using filters, joins, rules, aggregations, splits, unions and other common operations. • Natively process GoldenGate change records and keep live aggregates based on change records Monitor your Database Transactions • Analyze statistics of ongoing database activity • Identify hot records with many changes, monitor sensitive tables or records for activity and exceeding thresholds. • Correlate different transactions, for example confirm that a request is acknowledged. • Identify unusual or fraudulent activity, such as records that are created and soon after deleted Real-time BI Big Data Lakes Business Process Operational Dashboards OLTP Database GoldenGate Kafka - Oracle Enterprise Hub Oracle Stream Analytics How does my data do right now? Alert and Act on critical issues OLTP Database GoldenGate Kafka - Oracle Enterprise Hub Oracle Stream Analytics Target Database
- 38. Managed GoldenGate Service Copyright© 2019 Oracle and/or its affiliates. GoldenGate for OCI Marketplace GoldenGate Service (GGS) • Automated provisioning (via Terraform) • Customer managed software, runs on customer managed OCI Compute & Storage • Choose from 5 listings: GG Microservices, GG Classic, GG for Big Data, GG for Non-Oracle, or GG for Mainframe • Bring your own license (Processor, Term, Named User, Promotional…) • OCI native service (console-based, admin, monitoring, lifecycle, patching etc) • Oracle managed software (multi-tenant control plane, secure encryption keys and Trail Files deployed in customer tenancy) • Single, integrated customer experience for all aspects of GoldenGate functionality • Universal Cloud Credits
- 39. GoldenGate Service KeyUse Cases Copyright © 2019 Oracle and/or its affiliates. GoldenGate Service GoldenGate Service GoldenGate Service sync Database Use Cases Data Lake Use Cases HA/DR, Active-Active, Multi-Master, Zero-Downtime,DB Migrations Replicate transactions, data and DB events across OLTP DBs Trickle feed, load and stage data into Data Warehouse tables Streaming data ingestion to Object Storage, Hadoop, Kafka etc. Transform and process any data as it in arrives in the Stream Apply machine learning, geo- spatial and advanced rules GoldenGate Service GoldenGate Service GoldenGate Service ETL Pipeline AI/MLIoT Time Series push down
- 40. Industry and UseCases Oracle Stream Analytics Key Features Developer Experience Streaming Data Pipelines (ETL) with GoldenGate …for More Information Copyright © 2019 Oracle and/or its affiliates.
- 41. The preceding isintended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, timing, and pricing of any features or functionality described for Oracle’s products may change and remains at the sole discretion of Oracle Corporation. Statements in this presentation relating to Oracle’s future plans, expectations, beliefs, intentions and prospects are “forward-looking statements” and are subject to material risks and uncertainties. A detailed discussion of these factors and other risks that affect our business is contained in Oracle’s Securities and Exchange Commission (SEC) filings, including our most recent reports on Form 10-K and Form 10-Q under the heading “Risk Factors.” These filings are available on the SEC’s website or on Oracle’s website at http://www.oracle.com/investor. All information in this presentation is current as of September 2019 and Oracle undertakes no duty to update any statement in light of new information or future events. Safe Harbor Copyright © 2019 Oracle and/or its affiliates.
- 42. Oracle Stream Analytics 42Confidential– Oracle Internal/Restricted/Highly Restricted Integration with GoldenGate Kafka Cloud Services Contextual Data ML Models Real-time BI Big Data Lakes Business Process Operational Dashboards DB Events Ingest with GoldenGate Actions Oracle SQL Server MySQL IBM DB2 Z IBM DB2 i IBM DB2 LUW HP NonStop Informix Sybase Messaging Oracle GoldenGate Stage in Kafka Stream Analytics Capture Trail Files Delivery Trail Files
- 43. Pattern for LogicalData Zones & Topic Types 43Copyright © 2018, Oracle and/or its affiliates. All rights reserved. | Oracle OpenWorld 2018 Raw Data (LCR) Schema Events (DDL) Prepared Data Topics Master Data Topics ETL ETL 1 Topic : 1 Table Data Consumers Applications Analytics ODS (Data Store) Data Marts Data Warehouses Stream Data Producers Apps & DBs: Staging Trusted Master ETL Bulk Data Producers ETL Data Science Events are Pushed Batching Interactive Queries OLAP SQL Bucket 1 Bucket 2 Bucket 3