Downloaded 17 times
Uploaded byDatabricks
Cloud-native Semantic Layer on Data Lake
The document outlines a presentation by Dong Li, focusing on the challenges faced by a fast-growing SaaS company using AWS RDS for dashboards, including slow queries and difficulties scaling. It presents Apache Kylin as a solution for high-performance data analytics, highlighting its capabilities for OLAP data modeling, flexible dashboards, and lower latency. The roadmap for Apache Kylin's cloud-native analytics is also discussed, emphasizing the need for a unified semantic layer for improved data accessibility and management.
More Related Content
![[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...](https://cdn.slidesharecdn.com/ss_thumbnails/draganberic-lakehousearchitecturewithdeltalakeanddatabricks-221130080712-6e817e95-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Cloud-native Semantic Layer on Data Lake
![[Redis Released]- FalkorDB - Redis + Graph Agentic Memory’s Secret Sauce](https://cdn.slidesharecdn.com/ss_thumbnails/redisreleased-falkordbslidedeck-1125-251115194922-e1c0046b-thumbnail.jpg?width=640&height=640&fit=bounds)
Cloud-native Semantic Layer on Data Lake
- 1. Cloud-native Semantic Layeron Data Lake Dong Li PMC @ Apache Kylin, Founding Member @ Kyligence
- 2. About Me Dong Liis a Founding Member and Head of Product and Innovation at Kyligence, an Apache Kylin Core Developer (Committer) and member of the Project Management Committee (PMC) where he focuses on big data technology development. Previously, he was a Senior Engineer in eBay’s Global Analytics Infrastructure Department, a Software Development Engineer for Microsoft Cloud Computing and Enterprise Products.
- 3. Agenda ▪ Start froma real challenge ▪ The Solution ▪ Q&A
- 4. Customer Background • Afast-growing SaaS company in US • 1800 customers in 40+ countries • 1/3 Fortune 500 use • 8 Billion transactions per year • Dashboards for end users
- 5. Landscape & Challenges •Source Data in AWS RDS • Materialized views used for dashboards • Slow queries cost 5+ seconds • 4+ hours to refresh materialized views every day • Bottleneck at ~10 concurrent users • Couldn’t provide flexible dashboards • Number of views keeps increasing OLTP (RDS) OLAP (RDS) Materialized View Dashboard Export ETL SQL
- 6. Expectation for thefuture data platform • Flexible dashboards for end users • High performance (< 2s), high concurrency (> 100 users) • Easy to scale • Low data preparation latency (< 1 hour) • Flexible for new requirements • Enterprise-grade security: data recovery, row/column level access etc. • Totally on AWS • Low TCO • Open Platform for Machine learning, Internal Analytics etc.
- 7. Agenda ▪ Start froma real challenge ▪ The Solution ▪ Q&A
- 8. Apache Kylin: OpenSource Distributed Analytical Data Warehouse
- 9. Apache Kylin: ManagingYour Most Valuable Data • OLAP Data Modelling • Speed Up Analytics Using Pre-Calculation • ANSI SQL Interface • High Concurrency and High Performance • Batch & Streaming Together Presentation Visualization Big Data Platform Data Source Data Mart Hive Impala Spark SQL Kafka MapReduce …Spark
- 10. Apache Kylin Community& Adoptions 1000+ Global Adoptions Leading Open Source OLAP Github Stars JIRA Issues
- 11. Star Schema Benchmark Starschema benchmark: http://www.cs.umb.edu/~poneil/StarSchemaB.PDF 0 2 4 6 8 10 12 1.1 1.2 1.3 2.1 2.2 2.3 3.1 3.2 3.3 3.4 4.1 4.2 4.3 Latency(s) SSB Queries 条SQL响 Kylin SQL on Hadoop SQL Latency Lower is better 0 10 20 30 40 50 60 70 80 90 0 10 20 30 40 50 Latency(s) Data Scale 不同数据量性能 化 Kylin SQL on Hadoop Data Volume Scale Lower is better
- 12. Interactive Analysis withBI for Petabyte-Scale Datasets
- 13. select l_returnflag, o_orderstatus, sum(l_quantity) as sum_qty, sum(l_extendedprice)as sum_base_price from v_lineitem inner join v_orders on l_orderkey = o_orderkey where l_shipdate <= '1998-09-16' group by l_returnflag, o_orderstatus order by l_returnflag, o_orderstatus; Sort Aggr Filter TablesO(N) Join Parse SQL to an execution plan How Does Kylin Accelerate Queries? • Kylin uses Apache Calcite as the SQL parser and optimizer
- 14. How Does KylinAccelerate Queries? • Kylin optimizes and adapts the plan to an OLAP cube. • With less processing, Kylin can return the result instantly. Aggr Filter Tables Join Sort Sort Cube Filter Pick the best matched cube Rewrite toThese steps have already been completed in the cube build. O(1)
- 15. Apache Kylin BI ToolsApps Machine Learning SQL Runtime Workload Offload Workload Scan & filter Extract Load Architecture The architecture of Apache Kylin v4.0.0-alpha
- 16. Use Case: OnlineShopping Reporting The most visited website in Japan https://techblog.yahoo.co.jp/oss/apache-kylin/ § Our reporting system used Impala as a backend database previously. - It took a long time (about 60 sec) to show Web UI. § In order to lower the latency, we moved to Apache Kylin. - Average latency < 1sec for most cases Thanks to low latency with Kylin, we become possible to focus on adding functions for users. § We provide a reporting system that show statistics for store owners. - e. g. impressions, clicks and sales.
- 17. Apache Kylin 4.0Roadmap: Cloud Native Data analytics Apache Kylin Container Service (K8S, Docker) Interactive Reporting Dashboard OLAP ／ Data mart Resource Orchestration Data Lake Source file, Streams, Parquet on Object Storage (S3, ADSL) Metadata Security • Less Dependency, More Lightweight • Automated Scaling • Less Computing and Storage Cost • Automated DevOps
- 18. Data is nextOil The world’s most valuable resource is no longer oil, but data. —“The Economist” China’s Datasphere is expected to grow 30% on average over the next 7 years and will be the largest Datasphere of all regions by 2025 --IDC 175 Zettabytes By 2025 -- IDC
- 19. and, Data ismoving to Cloud
- 20. But, the ChaosHappens Again!!!
- 21. What is missinghere? ? ? ? Reporting Dashboard Ad-Hoc Data-as-a-Services Machine Learning EDW Datasets Data Lake Datasets Cloud Datasets Data Lake Application SQL / MDX Data Analysts Marketing User Operation Analysts
- 22. Unified Semantic Layer Govern DataPlatform Reporting Dashboard Ad-Hoc Data-as-a-Services Machine Learning Managed Datasets Managed KPIs CUSTOMER CUSTOMER NUMBER CUSTOMER NAME CUSTOMER CITY CUSTOMER POST CUSTOMER ST CUSTOMER ADDR CUSTOMER PHONE CUSTOMER FAX ORDER ORDER NUMBER ORDER DATE STATUS ORDER ITEM BACKORDERED QUANTITY ITEM ITEM NUMBER QUANTITY DESCRIPTION ORDER ITEM SHIPPED QUANTITY SHIP DATE Finance KPI ERP KPI Accounting KPI …… Marketing KPI Sales KPI EDW Datasets Data Lake Datasets Cloud Datasets Data Lake Application SQL / MDX One-stop Governed Platform • Data as a service • Single source of truth • Managed golden data Intelligent Data Platform • Machine Learning recommendation from SQL history • Optimizaed for PB data at scale • High performance and High Concurrency Analysts Delighted Platform • Supports most favorite BI tools • Support standard SQL/MDX • Reduce engineering efforts Data Analysts Marketing User Operation Analysts Intelligent Cubing managed data at scale
- 23. Unified Data-as-a-Service Layer UnifiedSemantic Layer
- 24. Use Case: LessCubes for More 2 Cubes vs 1200 IBM Cognos Cubes Challenge • 1200+ existing cubes to manage • 1000+ jobs to maintain • Time-to-insight over 4 days Job Job Job Job Job Job Job … … Job Job Job 1200+ IBM Cognos Cubes, 1000+ ETL Jobs with dependencies Merchant Topic Daily Cube Region Topic Daily Cube Merchant Topic Monthly Cube 2 Cubes, 1 ETL Job ┄ ┄ Agency Topic Daily Cube Agency Topic Monthly Cube Channel Topic Monthly Cube Region Topic Monthly Cube Merchant Topic Shanghai Cube Merchant Topic Beijing Cube Merchant Topic Zhejiang Cube Merchant Topic Guangdong Cube Sub scenarios Channel Topic Daily Cube Sub scenarios Sub scenarios Solution • Using Kylin replaced IBM Cognos backend but continue keep Cognos Reporting to interactive with Kyligence Result • 1000x improved maintenance efficiency • 10x faster and more stable analytics performance • Time-to-Insight less then 4 hours
- 25. Cloud-native Semantic Layeron Data Lake FinanceMarketingSales Index More… Data Lake Aggregation & Index ApplicationsSource Azure Blob Storage
- 26. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.