Download to read offline
Uploaded byAlluxio, Inc.
Introduction to Apache Iceberg™ & Tableflow
Low Latency Data Systems for Real-Time AI Meetup July 15, 2025 For more Alluxio Events: https://www.alluxio.io/events/ Speaker: - Tim Berglund (VP Developer Relations @ Confluent) In this talk, we'll build Iceberg up from the basics, see how the read and write path work, and explore how it supports streaming data sources like Apache Kafka™.
More Related Content
Similar to Introduction to Apache Iceberg™ & Tableflow
Introduction to Apache Iceberg™ & Tableflow
- 1. Tim Berglund Apache Icebergand Tableflow unifying the operational and analytical estates https://linkedin.com/in/tlberglund @tlberglund
- 2. …but how didwe get here? Iceberg is a format for expressing tabular data with ACID semantics at scale.
- 5.
- 7.
- 8. The Cloud DataWarehouse
- 9.
- 11. Let ’ s Build ApacheIceberg
- 12. Parquet files ✅ Columnar ✅Row groups ✅ Integrated metadata ❌ Not a database
- 13. Parquet files ina folder 🪣:/lakehouse orders 2024 2025 01 02 ✅ Similar files stored together ✅ Better than nothing ✅ Really quite scalable ❌ Still not a database
- 14. Parquet files ina folder 🪣:/lakehouse orders 2024 2025 01 02 ❌ No atomicity ❌ No isolation ❌ No consistency ❌ No actual schema
- 15. Parquet files ina folder 🪣:/lakehouse orders 2024 2025 01 02 ❌ No atomicity ❌ No isolation ❌ No consistency ❌ No actual schema No one will stop this ⬆
- 16. Parquet files ina folder 🪣:/lakehouse orders 2024 2025 01 02 ❌ The “table ” is just whatever files live in the folder at the moment.
- 17. Manifest File MANIFEST FILE ✅JSON file ✅ Column-level stats ✅ List of data files with arbitrary paths MANIFEST FILE ❌ Doesn ’ t handle multiple ingest events ✅ New definition of my “table ”
- 18. Manifest List MANIFEST FILE ✅JSON file ✅ Collection of Manifest Files, partitioning information MANIFEST FILE ✅ Really new definition of my “table ” MANIFEST LIST
- 19. Metadata File MANIFEST FILE ✅Another JSON file ✅ Collection of Manifest Lists, various metadata, and snapshots MANIFEST FILE ✅ This is it. This is what a table is. MANIFEST LIST METADATA FILE SNAPSHOT 0
- 20. Full Structure MANIFEST FILEMANIFEST FILE MANIFEST LIST MANIFEST FILE MANIFEST LIST METADATA FILE SNAPSHOT 1 SNAPSHOT 0
- 21. Full Structure MANIFEST FILEMANIFEST FILE MANIFEST LIST MANIFEST FILE MANIFEST LIST METADATA FILE SNAPSHOT 1 SNAPSHOT 0 CATALOG ✅ Catalog maps table name to metadata file snapshot table “orders“
- 22. Iceberg Catalog CATALOG ✅ JDBC ✅Any atomic datastore will do ✅ Maps table name to metadata file snapshot ✅ AWS Glue ✅ Hive Metastore ✅ Nessie ✅ REST Catalog
- 23. Write Path MANIFEST FILEMANIFEST FILE MANIFEST LIST MANIFEST FILE MANIFEST LIST METADATA FILE SNAPSHOT 1 SNAPSHOT 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 24. Write Path MANIFEST FILEMANIFEST FILE MANIFEST LIST MANIFEST FILE MANIFEST LIST CATALOG LAYER CATALOG METADATA LAYER DATA LAYER QUERY ENGINE 1⃣ Inbound write 2⃣ Get current metadata file 3⃣ Write data files 4⃣ Write manifest file 5⃣ Write manifest list METADATA FILE SNAPSHOT 0 METADATA FILE SNAPSHOT 0 SNAPSHOT 1 6⃣ Write metadata file with new snapshot
- 25. Write Path MANIFEST FILEMANIFEST FILE MANIFEST LIST MANIFEST FILE MANIFEST LIST CATALOG LAYER CATALOG METADATA LAYER DATA LAYER QUERY ENGINE 1⃣ Inbound write 2⃣ Get current metadata file 3⃣ Write data files 4⃣ Write manifest file 5⃣ Write manifest list 7⃣ Update/commit catalog METADATA FILE SNAPSHOT 0 METADATA FILE SNAPSHOT 0 SNAPSHOT 1 6⃣ Write metadata file with new snapshot
- 26. Read Path MANIFEST FILE MANIFESTLIST CATALOG LAYER CATALOG METADATA LAYER DATA LAYER QUERY ENGINE 1⃣ Inbound SELECT 2⃣ Get current metadata file 3⃣ Read metadata file 4⃣ Read manifest list 5⃣ Read manifest file METADATA FILE SNAPSHOT 0 SNAPSHOT 1 6⃣ Read data file(s)
- 27. Copy on Write 1:❤ 2:🔥 3:💩 4:🪣 5:🌱 6:🇦🇺 7:🏀 8:😳 MANIFESTFILE MANIFEST LIST METADATA FILE SNAPSHOT 0 SNAPSHOT 1 ❌ But we like updates ✅ Data files are immutable ✅ Suppose we have eight rows ✅ And we update row 5
- 28. Copy on Write 1:❤ 2:🔥 3:💩 4:🪣 5:🌱 6:🇦🇺 7:🏀 8:😳 MANIFESTFILE MANIFEST LIST METADATA FILE SNAPSHOT 1 SNAPSHOT 2 ❌ But we like updates ✅ Data files are immutable ✅ Suppose we have eight rows ✅ And we update row 5 5:😍 6:🇦🇺
- 29. Copy on Write(incl. snapshot 1 ) 1:❤ 2:🔥 3:💩 4:🪣 5:🌱 6:🇦🇺 7:🏀 8:😳 MANIFEST FILE MANIFEST LIST METADATA FILE SNAPSHOT 1 SNAPSHOT 2 ❌ But we like updates ✅ Data files are immutable ✅ Suppose we have eight rows ✅ And we update row 5 5:😍 6:🇦🇺 MANIFEST FILE MANIFEST LIST
- 30. Merge on Read 1:❤ 2:🔥 3:💩 4:🪣 5:🌱 6:🇦🇺 7:🏀 8:😳 MANIFESTFILE MANIFEST LIST METADATA FILE SNAPSHOT 0 SNAPSHOT 1 ❌ But we like updates ✅ Data files are immutable ❌ No, seriously, we like updates ✅ And we update row 5 again
- 31. Merge on Read 1:❤ 2:🔥 3:💩 4:🪣 5:🌱 6:🇦🇺 7:🏀 8:😳 MANIFESTFILE MANIFEST LIST METADATA FILE SNAPSHOT 1 SNAPSHOT 2 DEL 5 5:😍 ❌ But we like updates ✅ Data files are immutable ❌ No, seriously, we like updates ✅ And we update row 5 again ✅ We write a delete file, then the delta
- 32. Compute Engines ✅ Flink ✅Spark ✅ Dremio ✅ Databricks ✅ Snowflake
- 33.
- 34. Kafka Cluster Kafka Cluster Kafka Cluster Traditional Data LakeIngest Kafka Cluster Kafka Connect App App App
- 35. Kafka Cluster Kafka Cluster Kafka Cluster Traditional Data LakeIngest Kafka Cluster Kafka Connect App App App THE OLD WAY
- 36. Separated for generations Servethe needs of applications to transact with customers in real-time Support after-the-fact business analysis and reporting for various stakeholders OPERATIONAL ESTATE ANALYTICAL ESTATE
- 37. Streaming to datalakes is complicated STREAM INGEST PREP Type conversions Schema management Sync metadata to catalog Table maintenance Ingest Workflow Silver & Gold Tables Business-specific rules and logic CDC materialization Deduplication Filtering Bronze (raw) Tables Object Storage S3 GCS ABS Operational Data Analytical Data
- 38. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 39. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 40. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 41. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 42. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 43. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 44. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 45. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 46. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER
- 47. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 48. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 49. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 50. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 51. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 52. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 53. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 54. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 55. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1
- 56. Tableflow MANIFEST FILE MANIFEST LIST METADATAFILE SNAP 0 CATALOG LAYER CATALOG METADATA LAYER DATA LAYER MANIFEST FILE MANIFEST LIST SNAP 1 MANIFEST FILE MANIFEST LIST SNAP 2
- 60. Give Confluent Clouda try cnfl.io/cloudmeetupgift Get $400 worth of free credits for your first 30 Days. SKIP THE PAYWALL CONFLUENTDEV1 Promo Code Because you’re here + another $25 in Credits!