Download as PDF, PPTX
Uploaded byFlink Forward
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Julia Bennett, Netflix
The document discusses building event-time partitioning using Apache Flink and Apache Iceberg, specifically in the context of Netflix's playback data. It highlights the challenges of handling out-of-order and late-arriving events, as well as the implementation details of achieving exactly-once processing. Key trade-offs include managing low-latency outputs and complexity in failure recovery while optimizing for deduplication and partition efficiency.
More Related Content
Similar to Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Julia Bennett, Netflix
Streaming Event Time Partitioning with Apache Flink and Apache Iceberg - Julia Bennett, Netflix
- 1. Streaming Event-Time Partitioning WithApache Flink And Apache Iceberg Julia Bennett Flink Forward October 2019
- 2. What is thistalk about? ● Story of building event-time partitioning in Flink and Iceberg ○ Netflix playback data ○ Why event-time partitioning matters ○ Introduce Iceberg ○ Implementation details ○ Tradeoffs
- 4.
- 5. Processing Time: 2019-08-15, 11:53 EventTime: 2019-08-15, 9:17 Event Data 2019-08-15, 11:00 2019-08-15, 10:00 2019-08-15, 9:00 2019-08-15, 8:00 Table Partitioning Processing Time? Event Time?✔
- 6. Before: Playback Data Event Time Streaming(Near Real-Time) Batch (Hourly) Process Time Raw Events Session Summaries Business Logic **Lots of late events
- 7. Common Pattern Streaming (NearReal-Time) Batch (Hourly) Event Time Process Time Business Logic ?
- 8. Better Pattern? Streaming (NearReal-Time) Event Time RouterBusiness Logic ?
- 9. Why is thishard? ● Exactly once processing ○ Dedupe incoming stream ○ Append to table without duplicates (or loss) ● Out of order and late arriving events ○ Partition-level modifications ○ Efficiently modify long tail of partitions
- 10. Iceberg Sink ● ApacheIceberg (Incubating) is new table format developed at Netflix ○ Atomic commits and file-level changes ● Flink + Iceberg: Exactly once partition appends ○ Buffers records into partitioned files ○ Commits with each checkpoint
- 11. This should beeasy... Event Time Router .sink(Iceberg)
- 12. Problem: Too manyfiles Oh, and dedupe too (… e.g. 250 MILLION per day)
- 13. Source Dynamic Partitioner Traffic Metrics State Iceberg Sink Late Events State LateBatching Punctual Stream Dedupe Key State Dedupe Reduces # Files & Skew Offline Compaction
- 14. Dedupe Key State Dedupe Keyed Process Function +Timer Service Dynamic Partitioner Traffic Metrics State Side Output + Window + Broadcast Late Events State Late Batching Punctual Stream Keyed Process Function + Timer Service
- 15. After: Playback Data Streaming(Near Real-Time) Event Time RouterBusiness Logic Raw Events Session Summaries ~100K Files Per Day
- 16. Source Dynamic Partitioner Traffic Metrics State Iceberg Sink Offline Compaction Late Events State LateBatching Punctual Stream Dedupe Key State Dedupe -Generic event-time partitioning out of the box -
- 17. New Pattern Streaming (NearReal-Time) Event Time RouterBusiness Logic ?
- 18. Tradeoffs ● Low-latency streamingoutput ● Remove operation of offline batch job ● Failure recovery is hard ○ Multiple long-lived states ○ Writing directly to production table ● Complexity, even if abstracted away
- 19.
- 20.