Download as PDF, PPTX
More Related Content
Similar to Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
Migrating to Multi Cluster Managed Kafka - Conf42 - CloudNative
- 1. @NSilnitsky Migrating to MultiCluster Managed Kafka Migrating to a Multi-Cluster Managed Kafka with 0 Downtime Natan Silnitsky Backend Infra TL, Wix.com natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil
- 2. @NSilnitsky Migrating to MultiCluster Managed Kafka
- 3. @NSilnitsky Kafka in Wix Migratingto Multi Cluster Managed Kafka 2019 1 cluster, self hosted 5 K Topics > 45 K Partitions ~ 450 M Messages produced a day Kafka Broker
- 4. @NSilnitsky Migrating to MultiCluster Managed Kafka Kafka in Wix Today 2021 1 cluster, self hosted 20 K Topics > 200 K Partitions 2.5 B Messages produced a day
- 5. @NSilnitsky Migrating to MultiCluster Managed Kafka 2021 1 cluster, self hosted To multi cluster, managed Kafka platform So, migrate all this 20 K Topics > 200 K Partitions 2.5 B Messages produced a day overloaded 1. Better Cluster performance & flexibility 2. Transparent version upgrade 3. Easy to add a new Cluster 4. Tiered Storage
- 6. @NSilnitsky Migrating to MultiCluster Managed Kafka Wix wraps Kafka with Greyhound, a Scala/Java high-level SDK. ~2000 Wix microservices Kafka Producer Kafka Consumer Greyhound Producer Greyhound Consumer
- 7. @NSilnitsky Migrating to MultiCluster Managed Kafka Kafka Producer Greyhound Producer Greyhound Consumer Kafka Consumer Kafka Broker Checkout Service example Payments Service
- 8. Agenda 1. TheMulti Cluster (Kafka) 2. The Migration 3. What to Expect Migrating to Multi Cluster Managed Kafka
- 9. @NSilnitsky Kafka Brokers Cluster DC1 DC2 DC3DC4 Single Cluster Overload * intra-dc
- 10. @NSilnitsky DC1 DC2 DC3 DC4 Single Cluster Overload A A A A B KafkaCluster (by SLAs) B Multi Cluster C
- 11. @NSilnitsky DC1 DC2 DC3 DC4 Towhich Cluster? Kafka Cluster (by SLAs) A A A A B B C I want to produce a domain event.
- 12. @NSilnitsky aGreyhoundConsumerSpec(groupName, messageHandler, topicName) .ClusterA valproducer = GreyhoundBuilder.resilientProducer(topicName, _.ClusterB The Multi Cluster (Kafka)
- 13. @NSilnitsky Cluster A +DC1 Cluster A + DC2 The Multi Cluster (Kafka)
- 14. Agenda 1. TheMulti Cluster (Kafka) 2. The Migration 3. What to Expect Migrating to Multi Cluster Managed Kafka
- 15. @NSilnitsky Unbalanced brokers Unclear Kafkastrategy Too many partitions Real production impact → → → → Our Starting Point Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 16. @NSilnitsky Migrate on 0-drainTraffic? Migrations we ❤ Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 17. @NSilnitsky (Blockers) → Specific DCservices → Long time → Not gradual - Edge cases risk Q4 2020: CANCELED Migrate on 0-drain Traffic? Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 18. @NSilnitsky … HAS tobe Seamless & Production-safe. Migrate With Traffic! Multi Cluster To Single Cluster From overloaded self-hosted managed optimized
- 19. @NSilnitsky Automate, Automate, Automate. Multi Cluster To Single Cluster From overloadedself-hosted managed optimized … HAS to be Seamless & Production-safe.
- 20. Option B The Migration HTTP Service Greyhound Kafka Cluster Self-hosted DataCenter 1 Confluent Cloud @NSilnitsky
- 21. Option B The Migration Kafka Cluster Self-hosted DataCenter 1 Confluent Cloud Replicator service @NSilnitsky
- 22. Option B The Migration DataCenter 1 Topic 1 Partition 5 4 5 5 5 6 5 7 5 8 5 9 6 0 6 1 6 2 6 3 6 4 4 5 6 7 8 9 1 0 0 1 2 3 1. Consume 3. Save offset Mapping 2. Produce Confluent Cloud Replicator service Kafka Cluster Self-hosted @NSilnitsky
- 23. Option B The Migration Greyhound Kafka Cluster Self-hosted DataCenter 1 Replicator service Migration Orchestrator Confluent → @NSilnitsky
- 24. Option B The Migration Greyhound Kafka Cluster Self-hosted DataCenter 1 Replicator service Migration Orchestrator group of consumers Migrate Consumer: 1. Replicate to Confluent Confluent → @NSilnitsky
- 25. Greyhound Kafka Cluster Self-hosted Replicator service Migration Orchestrator Listen to events Option B TheMigration Subscribe Data Center 1 Unsubscribe Migrate Consumer: 1. Replicate to Confluent 2. Unsubscribe from self-hosted Kafka Cluster 3. Subscribe to Confluent 4. Rollback (Seek offset) success! Failure! @NSilnitsky Confluent →
- 26. @NSilnitsky The Migration Best Practices 1.Create a script that checks state by itself and stops if expected state is not reached. 2. Have a rollback readily available. 3. Start with test topics and no impact topics 4. Create custom metrics dashboards that show current state. The Migration
- 27. Agenda 1. TheMulti Cluster (Kafka) 2. The Migration 3. What to Expect Migrating to Multi Cluster Managed Kafka
- 28. @NSilnitsky Replicator service On-Prem Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic1 What to Expect when Migrating to Multi Cluster During Nov/Dec 2021
- 29. @NSilnitsky “ Unexpected error from SyncGroup: Theserver experienced an unexpected error when processing the request. Replicator service On-Prem Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic 1 What to Expect when Migrating to Multi Cluster During Nov/Dec 2021
- 30. @NSilnitsky What to Expectwhen Migrating to Multi Cluster kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=4194304 “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 31. @NSilnitsky What to Expectwhen Migrating to Multi Cluster Replicator service On-Prem Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic 1 It’s Christmas eve 🎄 2021. “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 32. @NSilnitsky What to Expectwhen Migrating to Multi Cluster kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=4194304 kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=8388608 “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 33. @NSilnitsky What to Expectwhen Migrating to Multi Cluster kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=4194304 kafka-configs.sh --bootstrap-server localhost:6667 --entity-type brokers --entity-default --alter --add-config message.max.bytes=8388608 Kafka records start getting DELETED faster than expected (for compact topics too) “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 34. @NSilnitsky What to Expectwhen Migrating to Multi Cluster Restored records from another Data Center Kafka records start getting DELETED faster than expected (for compact topics too) Luckily for us ... “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 35. @NSilnitsky What to Expectwhen Migrating to Multi Cluster Affected Broker versions: 1.1.0, 2.0.1, 2.1.1, 2.2.2, 2.4.0, 2.3.1 Fix: Changed dummy value for all topic configs “ Unexpected error from SyncGroup: The server experienced an unexpected error when processing the request.
- 36. @NSilnitsky Replicator service Self Hosted Kafka Cluster Managed Kafka Cluster Topic 2 Topic N Topic1 What to Expect when Migrating to Multi Cluster Topic 2 Topic N Topic 1 Sharded Consumers ... A B
- 37. @NSilnitsky Greyhound Kafka Cluster Self-hosted Replicator service Migration Orchestrator Unsubscribe What toExpect when Migrating to Multi Cluster We have the infra in place to control it from the outside! So we will soon be able to … 1. Switch Cluster 2. Skip Messages 3. Change processing rate Without GA! Confluent →
- 38. @NSilnitsky Migrating to MultiCluster Managed Kafka We used Greyhound & dedicated orchestration services for an automatic, safe, and gradual migration. Multi Cluster To Single Cluster From overloaded self-hosted Migrations we ❤ managed optimized
- 39.
- 40. @NSilnitsky A Scala/Java high-levelSDK for Apache Kafka. github.com/wix/greyhound Migrating to Multi Cluster Managed Kafka 0.2 is out!
- 41. @NSilnitsky Thank You! Migrating to MultiCluster Managed Kafka natansil.com twitter@NSilnitsky linkedin/natansilnitsky github.com/natansil 👉 slideshare.net/NatanSilnitsky Any questions?