Download as PDF, PPTX
Uploaded byGuozhang Wang
Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
The document discusses building large-scale streaming infrastructure across multiple data centers using Apache Kafka. It outlines the reasons for multi-data center architecture, design patterns, and the trade-offs between latency and consistency. Various options for data replication and management are presented, including active-active and active-passive replication strategies.
In this document
Powered by AI
This section discusses the necessity of multi-data center setups due to failures and geo-locality, along with challenges like bandwidth, latency, and consistency.
Explores different consistency models: Weak, Eventual, and Strong consistency with examples, stressing their implications for data integrity in distributed systems.
Examines the trade-offs between consistency and latency in WAN vs. LAN environments, setting the stage for the architecture design topics ahead.
Describes various strategies for data center deployment including Bunkerizing, Primary with Hot Standby, and Active-Active configurations to address failures.
Emphasizes the crucial role of ordering in distributed systems and introduces methods like vector clocks and Paxos for effective ordering.
Introduces Apache Kafka as a distributed messaging system that operates on a log-based message storage mechanism for efficient data handling.
Discusses how Kafka partitions logs across various machines, enhancing performance with configurable acknowledgment modes to handle failures.
Details the Active-Passive replication method in Kafka, its use for asynchronous data replication, and trade-offs including potential data loss.
Explains Active-Active replication in Kafka with a focus on global view maintenance and challenges during failovers with examples.
Addresses challenges of deploying Kafka across data centers, including differences in offsets, potential duplicates, and solutions for real-time applications.
Illustrates strategies for deploying Kafka in multiple data centers, focusing on multi-tenancy, security, and optimized latency.
Provides an example of EC2 multi-AZ deployment addressing considerations like latency, Zookeeper needs, and maintaining data integrity.
Wraps up the discussion with key takeaways about multi-DC trade-offs, inviting attendees to engage with Confluent for further learning.
More Related Content
Similar to Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
- 1. When One DataCenter is not Enough Guozhang Wang Strata San Jose, 2016 Building large-scale stream infrastructure across multiple data centers with Apache Kafka
- 2. 2 • Why acrossData Centers? • Design patterns for Multi-DC • Kafka for Multi-DC • Conclusion Agenda
- 3. 3 Why across DataCenters?
- 4. 4 Why across DataCenters • Catastrophic / expected failures • Routine maintenance • Geo-locality (Example: CDNs)
- 5. 5 Why NOT acrossData Centers • Low bandwidth (10Mbps - 1Gbps) • High latency (50ms - 450ms) • Much More $$$
- 6. 6 Why NOT acrossData Centers • … is hard and expensive
- 7. 7 Why NOT acrossData Centers • … is hard and expensive • … with real-time writes? Harder
- 8. 8 Why NOT acrossData Centers • … is hard and expensive • … with real-time writes? Harder • … consistently? Oh My!
- 9. 9 Consistency • Weak • Eventual •Strong Latency Guarantee
- 10. 10 Weak No Consistency •Now you see my writes, now you don’t • Best effort only, data can be stale • Examples: think of “caches”, VoIP
- 11. 11 Eventual Consistency • Youwill see my writes, … eventually • May need to resolve conflicts (manually) • Examples: think of “emails”, SMTP
- 12. 12 Strong Consistency • Youget what you write, for sure • External > Sequential > Causal (Session) • Examples: RDBMS, file systems
- 13. 13 • LAN: consistencyover latency • WAN: latency over consistency Latency vs. Consistency
- 14. 14 • Why acrossData Centers? • Design patterns for Multi-DC • Kafka for Multi-DC • Conclusion Agenda
- 15. 15 Option I: Don’tdo it • Bunkerize the single data center • Expect data loss at failures • Examples: ??
- 16. 16 Option II: Primarywith Hot Standby • Failover to hot standby (maybe inconsistent) • Window of data loss at failures • Examples: MySQL binlog
- 17. 17 Option III: Active-Active •Accepts writes in multi-DC • Resolve conflicts (strong / week consistency) • Examples: Amazon DynamoDB (vector clock) Google Spanner (2PC), Mesa (Paxos)
- 18.
- 19. 19 Ordering is Key •Vector clocks: partial ordering • Paxos, 2PC: global ordering • Log shipping: logical ordering (per-partition)
- 21. 21 Apache Kafka • Adistributed messaging system ..that store messages as a log!
- 22. 22 Store Messages asa Log 4 5 5 7 8 9 10 11 12... Producer Write Consumer1 Reads (offset 7) Consumer2 Reads (offset 10) Messages 3
- 23. 23 Partition the Logacross Machines Topic 1 Topic 2 Partitions Producers Producers Consumers Consumers Brokers
- 24. 24 ACK mode LatencyOn Failures “no" no network delay some data loss “leader" 1 network roundtrip a few data loss “all" ~2 network roundtrips no data loss Configurable ISR Commits
- 25. 25 • Why acrossData Centers? • Design patterns for Multi-DC • Kafka for Multi-DC • Conclusion Agenda
- 26. 26 Option I: Active-PassiveReplication Kafka local producers consumer consumer DC 1 MirrorMaker DC 2 Kafka replica
- 27. 27 Option I: Active-PassiveReplication • Async- replication across DC • May lose data on failover • Example: ETL to data warehouse / HDFS Kafka local producers consumer consumer DC 1 MirrorMaker DC 2 Kafka replica
- 28. 28 Option II: Active-ActiveReplication Kafka local Kafka aggregate Kafka aggregate producers producers consumer consumer MirrorMaker Kafka local on DC1 failure DC 1 DC 2
- 29. 29 Option II: Active-ActiveReplication • Global view on agg. cluster • Require offsets to resume • Example: store materialization, index updates Kafka local Kafka agg Kafka agg producers producers consumer consumer MirrorMaker Kafka local on DC1 failure DC 1 DC 2
- 30. 30 • Offsets notidentical between Kafka clusters • Duplicates during failover • Partition selection may be different • Solutions • Resume from log end offset (suitable for real-time apps) • Resume from a timestamp (ListOffsets, offset index: KIP-33) Caveats: offsets across DCs
- 31. 31 Option III: Deployacross DCs Kafka producers producers consumer consumer DC 1 DC 2
- 32. 32 Option III: Deployacross DCs • Multi-tenancy support • Security (0.9) • Quota Management (0.9) • Latency optimization • Rack-aware partition assignment (0.10) • Read affinity (future?) Kafka producers producers consumer consumer DC 1 DC 2
- 33. 33 • Same region:essentially same network • asymmetric partitioning is rare, low latency • Need at least 3 DCs for Zookeeper • Reserved instance to reduce churns • EIP for external clients, private IPs for internal communication • Reserved instance, local storage Example: EC2 multi-AZ Deployment
- 34. 34 Take-aways • Multi-DC: trade-offbetween latency and consistency • Kafka: replicated log streams for multihoming
- 35. Thank you Guozhang |guozhang@confluent.io | @guozhangwang Meet Confluent in booth #838 Confluent University ~ Kafka training ~ confluent.io/training Join the Stream Data Hackathon Apr 25, SF kafka-summit.org/hackathon/ Download Apache Kafka & Confluent Platform confluent.io/download