Download to read offline
Uploaded byKai Sasaki
Continuous Optimization for Distributed BigData Analysis
This document discusses challenges with distributed data analysis and Treasure Data's approach to addressing them. Some key points: - Distributed data analysis faces challenges around network bandwidth, throughput, data consistency, and reliability. - Treasure Data uses a columnar storage format based on MessagePack to more efficiently save bandwidth and storage space. - They implement time index pushdown to enable reading only relevant data within a time range, reducing network usage. - Automatic optimization of partitioning layout and repartitioning aims to balance partition file size, time ranges, and keys to maximize performance and throughput while minimizing memory pressure.
More Related Content
Similar to Continuous Optimization for Distributed BigData Analysis
Continuous Optimization for Distributed BigData Analysis
- 1. Continuous Optimization forDistributed BigData Analysis Kai Sasaki (Treasure Data)
- 2. Bio Kai Sasaki - SoftwareEngineer at Treasure Data - Hadoop, Presto - Apache Hivemall - Books 2
- 3.
- 4. Agenda - Who isTreasure Data - What is distributed data analysis? - What kind of challenges we have? - Our approach - Columnar Storage - Partitioning - Repartitioning 4
- 5.
- 6. Treasure Data • Foundedin Dec, 2011 • Mountain View, CA • DMP, CDP, IoT, Cloud • We joined Arm Oct, 2018 6
- 7. Treasure Data • OpenSource Lovers 7
- 8.
- 9. Arm x TreasureData • Pelion: Device-to-Device Platform 9
- 10. 10 Challenges based on OurExperience https://pixabay.com/en/adventure-height-climbing-mountain-1807524/
- 11. Distributed Data Analysis? •Large Scale Data • High Throughput • High Availability & Reliability • Data Consistency 11
- 12. Distributed Processing Engines •Hadoop • Presto • Spark 12
- 13. Typical Architecture • Master-Workermodel 13 https://www.tutorialspoint.com/apache_presto/apache_presto_architecture.htm
- 14. Distributed Plan 14 select t1.class, t2.features, count(1) from irist1 join iris t2 on t1.class = t2.class group by 1, 2;
- 15. Challenges • Network Bandwidth •Throughput • Transactional Processing • Data Consistency • System Reliability • Service Availability 15
- 16. Our Approach • ColumnarStorage • MessagePack based columnar format • Time Index Pushdown • Optimization of Partitioning Layout 16
- 17. Columnar Storage • Generaldesign for OLAP workload • Save IO bandwidth • Efficient compression and encoding • e.g. Parquet, ORC 17
- 18. MessagePack • JSON-like binaryserialization format • Faster and smaller • 100+ implementations • https://msgpack.org 18
- 19. MessagePack x ColumnarFile • Type embedded file format • Schema-on-Read • -> Saving network bandwidth and storage space efficiently 19
- 20. MessagePack x ColumnarFile 20
- 21. Time Index Pushdown •Read skipping by time range • Fitting to the typical analytical use cases • Saving network bandwidth 21
- 22. Time Index Pushdown •Indexed by PostgreSQL • Transactional Update • Data Consistency • GiST index achieves efficient multi column index 22
- 23.
- 24.
- 25. Partition Size? • Thepartition file size affects the performance significantly • 1000000 records / file • 256MB / file • But depends on the workload 25
- 26. Auto Optimization • Partitioninglayout should be fit to the actual workload • File size • Time range • Partitioning key 26
- 27. Repartitioning • Small distributedpartition files • High IO overhead • Few large partition files • High memory pressure TRADE OFF PROBLEM 27
- 28. Repartitioning • Partitioning keydecides the throughput • e.g. Customer segmentation by • User ID • Purchase item • Living address 28
- 29. User Defined Partitioning •Custom partitioning schema defined by our user side (or ourselves) 29
- 30.
- 31.
- 32.
- 33. User Defined Partitioning •Granularity • Partitioning Key Selection 33
- 34. Stella Connector • Repartitioning& UDP is designed as a Presto connector • Make use of Presto high scalability and reliability for such high workload 34
- 35. Stella Connector 35 CREATE TABLEremerged WITH (max_file_size = '256MB', max_time_range='48h') AS SELECT * FROM partition.sources WHERE table_schema = 'tpch_s1' AND table_name = 'lineitem' AND TD_TIME_RANGE(time, '1998-10-11', '1998-10-20')
- 36. Stella Connector • Scalable •Reliable • Easy to embed it into Workflow • Automatic Storage Optimization! 36
- 37. Recap - Treasure DataOverview - Architecture of Distributed Data Analysis - Challenges - Our Approach - Columnar Storage - Partitioning - Repartitioning 37
- 38.