Downloaded 809 times
Uploaded byconfluent
Microservices in the Apache Kafka Ecosystem
The document discusses the principles and practices of building event-driven microservices within the Kafka ecosystem, emphasizing the importance of using appropriate tools like Kafka, KStreams, and Kafka Connect for managing data streams and microservices architecture. It outlines ten key principles for effective streaming services, such as avoiding mutable state, decoupling publishers from subscribers, and leveraging stream processing for real-time data manipulation. The content highlights the need for a data-centric approach in modern software development to ensure scalability and fault tolerance amidst growing data complexity.
More Related Content
Similar to Microservices in the Apache Kafka Ecosystem
Microservices in the Apache Kafka Ecosystem
- 1. Ben Stopford, Engineer,Confluent
- 2. • Event DrivenMicroservices • The toolset: Kafka, KStreams, Connect • 10 Principals for Streaming Services What we’ll cover
- 3.
- 4.
- 5.
- 6. The Monolith Can wedo reuse, encapsulation?
- 7. The Monolith What happenswhen we grow?
- 8. Companies are inevitably acollection of applications They must work together to some degree
- 9. Inverse Conway Maneuver The'Inverse Conway Maneuver' recommends evolving your team and organizational structure to promote your desired architecture. Org Structure Software Architecture
- 10.
- 11.
- 12. But state mustinevitably be shared between services
- 13. Use a toolkitthat embraces decentralisation
- 15. Some simple patternsof distributed systems
- 16.
- 17. When do weneed Request Response? Looking things up What is in my shopping basket?
- 18. Event Driven Async /Fire and Forget / Brokered
- 19. Businesses are oftenmodeled as a sequence events. Add to cart Buy Make Payment Update Stock Email Conf Package Item Dispatch Fraud
- 20. When do weneed Event Driven?
- 21. SOA / Microservices MessageBroker Event Based Request/Response
- 22.
- 23.
- 24.
- 25.
- 26. As software engineerswe are inevitably affected by the tools we surround ourselves with. Languages, frameworks, even processes all act to shape the software we build.
- 27. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service The tools wechoose have a big effect on our architecture GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Event Driven Request Response
- 28. Kafka is wellsuited to Event Driven Architectures It will lead you down that path
- 29.
- 30.
- 31. What is aDistributed Log?
- 32. Shard on theway in Producing Services Kafka Consuming Services
- 33. Each shard isa queue Producing Services Kafka Consuming Services
- 34.
- 35. Reduces to aglobally ordered queue
- 36.
- 37.
- 38. Build ‘Always On’Services Rely on Fault Tolerant Broker
- 39. Services can “Rewind& Replay” the log Rewind & Replay
- 40. Compacted Log (retains onlylatest version) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 41. A database enginefor data-in-flight
- 42. Max(price) From orders where ccy=‘GBP’ over1 day window emitting every second Continuously Running Queries
- 43. What is streamprocessing engine? Data Index Query Engine Query Engine vs Database Finite source Stream Processor Infinite source
- 44. Windowing For unordered orunpredictable streams Sliding Fixed (tumbling)
- 45. Features: similar to databasequery engine JoinFilter Aggr- egate View Window
- 46. Stateful Stream Processing stream Compacted stream Join StreamData Stream-Tabular Data Windowed Stream Locally Cached Table (disk resident) KafkaKafka Streams
- 47.
- 48.
- 49.
- 50.
- 51. Sometimes you needto physically move data
- 52. Replicate it, soboth copies are identical
- 53.
- 54.
- 55.
- 56. Service Backbone Scalable, FaultTolerant, Concurrent, Strongly Ordered, Stateful
- 57. Embeddable tool fordata manipulation Adapt data streams (transformation, stream maintenance, analytic function)
- 58. Replicate Data SourcesExactly Create Regenerable, Streaming Views
- 59. How do weactually do this? 10 (opinionated) principals for Streaming Services
- 60. 1. Don’t useKafka for shopping carts! (OK, you can, but use sparingly) Shopping Cart UI Service Broker/durability/broadcast add little to request response
- 61. Do use Kafkafor your core business processing. Think “business events”. An order was created, a payment was received, a trade was booked etc. Pub/Sub. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
- 62. 2. Pick Topicswith Business Significance Orders Payments Returns Invoices
- 63. Give your messagesmeaningful IDs and version them OrdersService1-Order-1234-v2 Should relate to the real world Should be Versioned (if mutable) Include the service name Note the key used for sharding in Kafka may not be this key
- 64. 3. Decouple publishersfrom subscribers GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service
- 65. Add Request/Response only whereneeded GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service REST
- 66. 4. Use thelog to regenerate state Avoid journaling incoming events
- 67. Event Source sideeffects • Use offsets to tie these back to the stream • Store in: • Kafka • Kstreams state store • Other DB
- 68. 5. Apply theSingle Writer Principal • Change at source (by calling that service) • Let the change propagate back • Keep local copies read only. GUI UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service (1) Change Orders at source (2) Let the change propagate through
- 69. 6. Leverage keeping datasetsinside the broker customer country supplier ex- rates
- 70. Leverage keeping onlythe latest version (table view) Version 3 Version 2 Version 1 Version 2 Version 1 Version 5 Version 4 Version 3 Version 2 Version 1
- 71. Join & Processon the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
- 72. 7. Prefer streamprocessing over maintaining historic views UI Service Orders Service Returns Service Fulfilment Service Payment Service Stock Service Orders Historic “copies” diverge
- 73. Join & Processon the fly stream Compacted stream Join Orders Customers KafkaKafka Streams
- 74. 8. Sometimes youneed historic views. => Replicate & Keep Read Only
- 75.
- 76.
- 77.
- 78. 9. Use Schemas (especiallyif data is retained)
- 79.
- 80.
- 81. Confluent Schema Registry canhelp Orders Service Schema Registry Email Service Avro
- 82. 10. Consider “Stream Management”Services Stream Management Kafka • Retaining data => Admin tasks • Similar to the role of a DBA • Data Migration • Repartitioning • Latest/versioned • Environment Management • CQRS
- 83. KStreams is agood toolset for this
- 84.
- 85. So… 1. Don’t useKafka for shopping carts! 2. Pick Topics with Business Significance 3. Decouple publishers from subscribers 4. Use the log to regenerate state 5. Apply the Single Writer Principal 6. Leverage keeping datasets inside the broker 7. Prefer stream processing over maintaining historic views 8. Sometimes you need historic views. => Replicate Read Only 9. Use Schemas 10. Consider “Stream Management” Services
- 86. Microservices push usaway from shared, mutable state
- 87. But state needsto be communicated
- 88. In an increasinglydata-heavy world we need tools to do this efficiently
- 89. …and in realtime.
- 90. We need adata-centric toolset to do this All Your Data Immutable Streams Event Driven Services Stream Management Services Legacy CDC View Replication Polyglotic Persistence tables streams Streaming Services
- 91. Keep it simple, Keepit moving
- 92.
Editor's Notes
- #4 Today we’re going to talk about two fields which sit somewhat apart. Stream Processing & Business Systems Services encourage us to share responsibilities. To rely on others. To collaborate and evolve a business’s view of the world over time. A world that is inherently decentralised. Inherently spread across many interconnected systems. Spreading many disparate subsets of our business’s state. Yet our approach rarely reflects this. We tend to think in centralised ways, we accumulate data. We protect it. We control it. We hoard it. But it does not need to be this way.
- #6 Today we’re going to talk about two fields which sit somewhat apart. Stream Processing & Business Systems Services encourage us to share responsibilities. To rely on others. To collaborate and evolve a business’s view of the world over time. A world that is inherently decentralised. Inherently spread across many interconnected systems. Spreading many disparate subsets of our business’s state. Yet our approach rarely reflects this. We tend to think in centralised ways, we accumulate data. We protect it. We control it. We hoard it. But it does not need to be this way.
- #15 My name is Ben Stopford. I work as an engineer on Apache Kafka. In a previous life I did the most centralised thing you could possibly do. I built a single, central database for a large financial institution. This talk is about exploring the exact opposite approach to this problem. It’s about thinking of the world in terms of the evolution and provisioning of state across many systems. It’s about embracing decentralisation. It’s about thinking in streams.