![© 2017 Dremio Corporation @DremioHQ
Columnar data
persons = [{
name: ’Joe',
age: 18,
phones: [
‘555-111-1111’,
‘555-222-2222’
]
}, {
name: ’Jack',
age: 37,
phones: [ ‘555-333-3333’ ]
}]](https://image.slidesharecdn.com/stratasanjose2017parquetarrow-170731224800/75/The-Future-of-Column-Oriented-Data-Processing-With-Apache-Arrow-and-Apache-Parquet-26-2048.jpg){kind=icon}
![© 2017 Dremio Corporation @DremioHQ
Record Batch Construction
Schema
Negotiation
Dictionary
Batch
Record
Batch
Record
Batch
Record
Batch
name (offset)
name (data)
age (data)
phones (list offset)
phones (data)
data header (describes offsets into data)
name (bitmap)
age (bitmap)
phones (bitmap)
phones (offset)
{
name: ’Joe',
age: 18,
phones: [
‘555-111-1111’,
‘555-222-2222’
]
}
Each box (vector) is contiguous memory
The entire record batch is contiguous on wire](https://image.slidesharecdn.com/stratasanjose2017parquetarrow-170731224800/75/The-Future-of-Column-Oriented-Data-Processing-With-Apache-Arrow-and-Apache-Parquet-27-2048.jpg){kind=icon}
![© 2017 Dremio Corporation @DremioHQ
Columnar data
persons = [{
name: ’Joe',
age: 18,
phones: [
‘555-111-1111’,
‘555-222-2222’
]
}, {
name: ’Jack',
age: 37,
phones: [ ‘555-333-3333’ ]
}]](https://crownmelresort.com/image.slidesharecdn.com/stratasanjose2017parquetarrow-170731224800/75/The-Future-of-Column-Oriented-Data-Processing-With-Apache-Arrow-and-Apache-Parquet-26-2048.jpg){kind=icon}
![© 2017 Dremio Corporation @DremioHQ
Record Batch Construction
Schema
Negotiation
Dictionary
Batch
Record
Batch
Record
Batch
Record
Batch
name (offset)
name (data)
age (data)
phones (list offset)
phones (data)
data header (describes offsets into data)
name (bitmap)
age (bitmap)
phones (bitmap)
phones (offset)
{
name: ’Joe',
age: 18,
phones: [
‘555-111-1111’,
‘555-222-2222’
]
}
Each box (vector) is contiguous memory
The entire record batch is contiguous on wire](https://crownmelresort.com/image.slidesharecdn.com/stratasanjose2017parquetarrow-170731224800/75/The-Future-of-Column-Oriented-Data-Processing-With-Apache-Arrow-and-Apache-Parquet-27-2048.jpg){kind=icon}
Uploaded byDremio Corporation
The Future of Column-Oriented Data Processing With Apache Arrow and Apache Parquet
The document discusses the advancements and benefits of the column-oriented data processing frameworks Apache Arrow and Apache Parquet, highlighting their roles in enhancing interoperability and performance in data analytics. Key benefits include significant speed increases in data processing and efficient memory usage, along with the ongoing involvement of various development teams from major open source projects. Future plans include integrations and enhancements across various systems to maximize efficiency and capabilities for users.
More Related Content
Similar to The Future of Column-Oriented Data Processing With Apache Arrow and Apache Parquet
The Future of Column-Oriented Data Processing With Apache Arrow and Apache Parquet
- 1. © 2017 DremioCorporation @DremioHQ The future of column-oriented data processing with Arrow and Parquet Jacques Nadeau, CTO Dremio, VP Apache Arrow Julien Le Dem, Principal Architect Dremio, VP Apache Parquet
- 2. © 2017 DremioCorporation @DremioHQ • CTO of Dremio • Apache member • VP Apache Arrow • Apache PMCs: Arrow, Calcite, Drill, Incubator Julien Le Dem @J_ Jacques Nadeau @intjesus • Principal Architect at Dremio • Formerly Tech Lead at Twitter on Data Platforms. • Creator of Parquet • Apache member • Apache PMCs: Arrow, Incubator, Kudu, Pig, Parquet
- 3. © 2017 DremioCorporation @DremioHQ Agenda • Community Driven Standard • Interoperability and Ecosystem • Benefits of Columnar representation – On disk (Apache Parquet) – In memory (Apache Arrow) • Future of columnar
- 4. © 2017 DremioCorporation @DremioHQ Community Driven Standard
- 5. © 2017 DremioCorporation @DremioHQ An open source standard • Parquet: Common need for on disk columnar. • Arrow: Common need for in memory columnar. • Arrow building on the success of Parquet. • Benefits: – Share the effort – Create an ecosystem • Standard from the start
- 6. © 2017 DremioCorporation @DremioHQ The Apache Arrow Project • New Top-level Apache Software Foundation project – Announced Feb 17, 2016 • Focused on Columnar In-Memory Analytics 1. 10-100x speedup on many workloads 2. Common data layer enables companies to choose best of breed systems 3. Designed to work with any programming language 4. Support for both relational and complex data as-is • Developers from 13+ major open source projects involved – A significant % of the world’s data will be processed through Arrow! Calcite Cassandra Deeplearning4j Drill Hadoop HBase Ibis Impala Kudu Pandas Parquet Phoenix Spark Storm R
- 7. © 2017 DremioCorporation @DremioHQ Interoperability and Ecosystem
- 8. © 2017 DremioCorporation @DremioHQ Shared Need => Open Source Opportunity “We are also considering switching to a columnar canonical in-memory format for data that needs to be materialized during query processing, in order to take advantage of SIMD instructions” -Impala Team “A large fraction of the CPU time is spent waiting for data to be fetched from main memory…we are designing cache-friendly algorithms and data structures so Spark applications will spend less time waiting to fetch data from memory and more time doing useful work” – Spark Team - PySpark Integration: 53x speedup (IBM spark work on SPARK-13534) http://s.apache.org/arrowstrata1 - Streaming Arrow Performance 7.75GB/s data movement http://s.apache.org/arrowstrata2 - Arrow Parquet C++ Integration 4GB/s reads http://s.apache.org/arrowstrata3 - Pandas Integration 9.71GB/s http://s.apache.org/arrowstrata4
- 9. © 2017 DremioCorporation @DremioHQ Before With Arrow Impala DremioPandas Spark Impala DremioPandas Spark
- 10. © 2017 DremioCorporation @DremioHQ Benefits of Columnar formats @EmrgencyKittens
- 11. © 2017 DremioCorporation @DremioHQ Columnar layout Logical table representation Row layout Column layout
- 12. © 2017 DremioCorporation @DremioHQ On Disk and in Memory • Different trade offs – On disk: Storage. • Accessed by multiple queries. • Priority to I/O reduction (but still needs good CPU throughput). • Mostly Streaming access. – In memory: Transient. • Specific to one query execution. • Priority to CPU throughput (but still needs good I/O). • Streaming and Random access.
- 13. © 2017 DremioCorporation @DremioHQ Parquet on disk columnar format
- 14. © 2017 DremioCorporation @DremioHQ Parquet on disk columnar format • Nested data structures • Compact format: – type aware encodings – better compression • Optimized I/O: – Projection push down (column pruning) – Predicate push down (filters based on stats)
- 15. © 2017 DremioCorporation @DremioHQ Access only the data you need a b c a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5 a b c a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5 a b c a1 b1 c1 a2 b2 c2 a3 b3 c3 a4 b4 c4 a5 b5 c5 + = Columnar Statistics Read only the data you need!
- 16. © 2017 DremioCorporation @DremioHQ Parquet nested representation Document DocId Links Name Backward Forward Language Url Code Country Columns: docid links.backward links.forward name.language.code name.language.country name.url Borrowed from the Google Dremel paper https://blog.twitter.com/2013/dremel-made-simple-with-parquet
- 17. © 2017 DremioCorporation @DremioHQ Arrow in memory columnar format
- 18. © 2017 DremioCorporation @DremioHQ Arrow goals • Well-documented and cross language compatible • Designed to take advantage of modern CPU characteristics • Embeddable in execution engines, storage layers, etc. • Interoperable
- 19. © 2017 DremioCorporation @DremioHQ Arrow in memory columnar format • Nested Data Structures • Maximize CPU throughput – Pipelining – SIMD – cache locality • Scatter/gather I/O
- 20. © 2017 DremioCorporation @DremioHQ CPU pipeline
- 21. © 2017 DremioCorporation @DremioHQ Minimize CPU cache misses a cache miss costs 10 to 100s cycles depending on the level
- 22. © 2017 DremioCorporation @DremioHQ Focus on CPU Efficiency Traditional Memory Buffer Arrow Memory Buffer • Cache Locality • Super-scalar & vectorized operation • Minimal Structure Overhead • Constant value access – With minimal structure overhead • Operate directly on columnar compressed data
- 23. © 2017 DremioCorporation @DremioHQ Arrow Messages, RPC & IPC
- 24. © 2017 DremioCorporation @DremioHQ Common Message Pattern • Schema Negotiation – Logical Description of structure – Identification of dictionary encoded Nodes • Dictionary Batch – Dictionary ID, Values • Record Batch – Batches of records up to 64K – Leaf nodes up to 2B values Schema Negotiation Dictionary Batch Record Batch Record Batch Record Batch 1..N Batches 0..N Batches
- 25. © 2017 DremioCorporation @DremioHQ Language Bindings Parquet • Target Languages – Java – CPP – Python & Pandas • Engines integration: – Many! Arrow • Target Languages – Java – CPP, Python – R (underway) – C, Ruby, JavaScript • Engines integration: – Drill – Pandas, R – Spark (underway)
- 26. © 2017 DremioCorporation @DremioHQ Columnar data persons = [{ name: ’Joe', age: 18, phones: [ ‘555-111-1111’, ‘555-222-2222’ ] }, { name: ’Jack', age: 37, phones: [ ‘555-333-3333’ ] }]
- 27. © 2017 DremioCorporation @DremioHQ Record Batch Construction Schema Negotiation Dictionary Batch Record Batch Record Batch Record Batch name (offset) name (data) age (data) phones (list offset) phones (data) data header (describes offsets into data) name (bitmap) age (bitmap) phones (bitmap) phones (offset) { name: ’Joe', age: 18, phones: [ ‘555-111-1111’, ‘555-222-2222’ ] } Each box (vector) is contiguous memory The entire record batch is contiguous on wire
- 28. © 2017 DremioCorporation @DremioHQ Moving Data Between Systems RPC • Avoid Serialization & Deserialization • Layer TBD: Focused on supporting vectored io – Scatter/gather reads/writes against socket IPC • Alpha implementation using memory mapped files – Moving data between Python and Drill • Working on shared allocation approach – Shared reference counting and well-defined ownership semantics
- 29. © 2017 DremioCorporation @DremioHQ RPC: Single system execution The memory representation is sent over the wire. No serialization overhead. Scanner Scanner Scanner Parquet files projection push down read only a and b Partial Agg Partial Agg Partial Agg Agg Agg Agg Shuffle Arrow batches Result
- 30. © 2017 DremioCorporation @DremioHQ Multi-system IPC SQL engine Python process User defined function SQL Operator 1 SQL Operator 2 reads reads
- 31. © 2017 DremioCorporation @DremioHQ Summary and Future
- 32. © 2017 DremioCorporation @DremioHQ Current activity: • Spark Integration (SPARK-13534) • Dictionary encoding (ARROW-542) • Time related types finalization (ARROW-617) • Bindings: – C, Ruby (ARROW-631) –JavaScript (ARROW-541)
- 33. © 2017 DremioCorporation @DremioHQ What’s Next • Arrow RPC/REST – Generic way to retrieve data in Arrow format – Generic way to serve data in Arrow format – Simplify integrations across the ecosystem
- 34. © 2017 DremioCorporation @DremioHQ RPC: arrow based storage interchange The memory representation is sent over the wire. No serialization overhead. Scanner projection/predicate push down Operator Arrow batches Storage Mem Disk SQL execution Scanner Operator Scanner Operator Storage Mem Disk Storage Mem Disk …
- 35. © 2017 DremioCorporation @DremioHQ RPC: arrow based cache The memory representation is sent over the wire. No serialization overhead. projection push down Operator Arrow-based Cache SQL execution Operator Operator …
- 36. © 2017 DremioCorporation @DremioHQ What’s Next • Parquet – Arrow Nested support for Python & C++ • Arrow IPC Implementation • Kudu – Arrow integration • Apache {Spark, Drill} to Arrow Integration – Faster UDFs, Storage interfaces • Support for integration with Intel’s Persistent Memory library via Apache Mnemonic
- 37. © 2017 DremioCorporation @DremioHQ Get Involved • Join the community – dev@{arrow,parquet}.apache.org – Slack: • https://apachearrowslackin.herokuapp.com/ – http://{arrow,parquet}.apache.org – Follow @Apache{Parquet,Arrow}