confluent, profile picture
Uploaded byconfluent
PDF, PPTX237 views

Unlocking value with event-driven architecture by Confluent

The document outlines a presentation on event-driven architecture (EDA) and its benefits for businesses, emphasizing how EDA can enhance real-time data processing and decision-making through various applications and use cases. It discusses the transition from monoliths to event-driven microservices, highlighting the advantages of loose coupling, fault tolerance, and cost-effectiveness, along with addressing challenges such as ordering and complexity. The document also introduces Confluent's data streaming platform as a solution for managing EDA and streamlining operational efficiency across various industries.

Embed presentation

Download as PDF, PPTX
1 / 106
2 / 106
3 / 106
4 / 106
Most read
5 / 106
6 / 106
7 / 106
8 / 106
9 / 106
10 / 106
11 / 106
12 / 106
Most read
13 / 106
14 / 106
15 / 106
16 / 106
17 / 106
18 / 106
19 / 106
20 / 106
21 / 106
22 / 106
Most read
23 / 106
24 / 106
25 / 106
26 / 106
27 / 106
28 / 106
29 / 106
30 / 106
31 / 106
32 / 106
33 / 106
34 / 106
35 / 106
36 / 106
37 / 106
38 / 106
39 / 106
40 / 106
41 / 106
42 / 106
43 / 106
44 / 106
45 / 106
46 / 106
47 / 106
48 / 106
49 / 106
50 / 106
51 / 106
52 / 106
53 / 106
54 / 106
55 / 106
56 / 106
57 / 106
58 / 106
59 / 106
60 / 106
61 / 106
62 / 106
63 / 106
64 / 106
65 / 106
66 / 106
67 / 106
68 / 106
69 / 106
70 / 106
71 / 106
72 / 106
73 / 106
74 / 106
75 / 106
76 / 106
77 / 106
78 / 106
79 / 106
80 / 106
81 / 106
82 / 106
83 / 106
84 / 106
85 / 106
86 / 106
87 / 106
88 / 106
89 / 106
90 / 106
91 / 106
92 / 106
93 / 106
94 / 106
95 / 106
96 / 106
97 / 106
98 / 106
99 / 106
100 / 106
101 / 106
102 / 106
103 / 106
104 / 106
105 / 106
106 / 106
Unlocking Value With Event-Driven Architecture By Confluent 1
9:30 - 10:00 10:00 - 10:15 10:15 - 11:15 11:15 - 11:30 11:30 - 12:30 12.30 - 13:00 Welcome Coffee SKY verso l’Event-Driven Architecture Daniele De Angelis, Senior Specialist – Architecture - Sky Italia Luca Cogliandro, Senior Manager – Software Engineering Integration - Sky Italia Sessione 1: How Event-Driven Architectures Enable Business Value Samuele Dell'Angelo, Senior Solutions Engineer - Confluent Break Sessione 2: Monoliths to Event-Driven Microservices with Confluent Samuele Dell'Angelo, Senior Solutions Engineer - Confluent Networking Light Lunch Agenda
SKY verso l’Event-Driven Architecture Daniele De Angelis Senior Specialist – Architecture,Sky Italia Luca Cogliandro Senior Manager – Software Engineering Integration, Sky Italia 3
API Exposure Adapter GET Composite Service Asyncrhonous Dispositive Common services Proactive Monitoring (shift-left) Adapter POST Fast Information Cloud/Paas Infrastructure Customer Touchpoint (LightHouse) Event Driven Architecture STEP 1 Data Driven STEP 3 STEP 2 Guidelines REST Api Design Pass Through Design Business Logic Orchestration Choreography and Event sourcing Application Design Error Handling and Resubmission Logging and monitoring Pass Through GET & POST (synchronous API) Logging Every Hermes Enterprise Framework pillar is based on a set of Enterprise and Integration Guidelines. These pillars translate into a set of Design Patterns to be used within the framework design and implementation process CI/CD pipelines
Event-driven architecture is defined as a software design pattern in which decoupled applications can asynchronously publish and subscribe to events via an event broker Event messages are published to a message broker, and interested subscribers or services can consume these events Participants exchange events without a centralized point of control. Each local transaction publishes domain events that trigger local transactions in other services. Benefits ➢ Loose coupling ➢ Fault tolerance ➢ Cost-effective Challenges ➢ Ordering ➢ Idempotency ✔ Consents ✔ RTD KPIs Benefits ➢ Distributed transaction ➢ Rollback operation ➢ Local transactions isolation Challanges ➢ Complexity ➢ Resilient implementation ➢ Observability ✔ Order creation Message broker Service A (Local transaction) Service B (Local transaction) Service C (Local transaction) Msg Input A Msg Input B Msg Input C Msg Output A Msg Output B Msg Output C
Event-driven Architecture based on Kafka Confluent Cloud • Platform PaaS • IAC CICD Automation • Production Cluster Multi Zone • Test Cluster Single Zone Confluent Platform Data Platform RTD • Topic Name Strategy 🡪 Business Domain • Schema Registry 🡪 Data Governance Producer Consumer Service Exposure • KPI ✔ 4 Million events produced and consumed in 10 minutes ✔ Live update major KPI Tracking Tools Tibco Data Platform Consents KPI events events AWS • Infrastructural • Consumer Lag • DLT not empty • No events into Topic
How Event-Driven Architectures Enable Business Value Samuele Dell’Angelo Senior Solution Engineer, Confluent 7
01 02 03 04 05 06 Intro to Event Streaming: What is Event Streaming? Data: The New Currency of Business Why you need Data Streaming Why Confluent Why Data Streaming for Data Products and AI Customer Stories and use cases Agenda
Intro to Event Streaming
Software is Changing every Aspect of life 10
Batch processing Siloed Slow Real-time stream processing Connected Fast OLD WAY NEW WAY Today, Software Is the Business
What Are Events Transportation TPMS sensor in Carol’s car detected low tire-pressure at 5:11am. Kafka Banking Alice sent $250 to Bob on Friday at 7:34pm. Kafka Retail Sabine’s order of a Fujifilm camera was shipped at 9:10am. Kafka
The Shift in Thinking: Event Streams 13 Rich User experiences Real-time events Real-time Event Streams A Sale A shipment A Trade Customer Actions Data driven operation Real-time Analytics Microservices Stream Processing
Data: The New Currency of Business Success
Data is the Fuel for a Modern Business Loyalty Rewards Payments Processing Real-time Trades Fraud Detection Personalized Recommendations
Unlocking Data and Analytics is Much Harder than it Should Be
17 There are Two Main Locations for Data OPERATIONAL ESTATE Serve the needs of applications to transact with customers in real-time ANALYTICAL ESTATE Support after-the-fact business analysis and reporting for various stakeholders
OPERATIONAL ESTATE Serve the needs of applications to transact with customers in real-time ANALYTICAL ESTATE Support after-the-fact business analysis and reporting for various stakeholders 18 As Data Needs Multiplied, Data Complexity Escalated
… creating this giant mess!
Your teams are building applications on this messy data architecture Recommendations Real-time payments Branch monitoring Real-time bot detection Real-time alerts Live customer chatbot Omni-channel orders Real-time inventory Mobile banking Fraud detection
Why you need Data Streaming
Data Streaming: A New Paradigm to Set Data in Motion Continuously ingest, process, govern and share streams of data in real-time Real-time Data A Transfer A Login A Trade A Customer Experience Rich Front-End Customer Experiences Real-Time Backend Operations New Products and Services “We need to shift our thinking from everything at rest to everything in motion.”
From Giant Mess to Central Nervous System DATA STREAMING PLATFORM “For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income.”
Shift Your Thinking Point-in-time queries on static data Continuous processing at generation time Tightly coupled point-to-point integrations Multi-subscriber decoupled architecture Producer-presented, governed & reusable data Complex data wrangling downstream Batch-based data integration Event-driven continuous data flows
Apply a Data Product Thinking FINANCIAL SERVICES RETAIL & E-COMMERCE MANUFACTURING HEALTHCARE TRANSPORTATION GAMING Customers Accounts Web logins Clickstream Transactions Payments Branches Claims Trades Customers Accounts Purchases Web clickstream Inventory Stores Employees Shipments Machine Telemetry Inventory Orders Shipments Suppliers Assets Equipment Patients Accounts Diagnoses Treatments Claims Readings Providers Players Achievements Profiles Games Purchases Vehicle Telemetry Drivers Customers Conditions Deliveries Repairs Insurers Chats High Scores Data products: Real-time, discoverable, contextualized, trustworthy and reusable
Use Data Products to Build New Business Applications Companies that treat data like a product can: ● Reduce the time it takes to implement new use cases by as much as 90% ● Decrease total ownership costs by up to 30% (technology, development, and maintenance) ● Reduce their risk and data governance burden Customers Accounts Web Logins Transactions Fraud Detection Customers Accounts Inventory Purchases Web Clickstream Customer Loyalty Vehicle Telemetry Repairs Drivers Customers Delivery Management Data Products Use Cases
Build an Effective Data Flywheel Strategy by Reusing Data Products Across Multiple Use Cases Supply chain systems Watch lists Profile mgmt Incident mgmt Customers Threat vector Customer churn Transactions Payments Mainframe data Inventory Weather Telemetry IoT data Notification engine Payroll systems Distribution Order fulfillment CRM systems Mobile application Personalization Clickstreams Customer loyalty Change logs Purchases Recommendation engine Web application Payment service Heatmap service ITSM systems Central log systems (CLS) Fraud & SIEM systems Alerting systems AI/ML engines Visualization apps
PROCESS GOVERN CONNECT STREAM Accounts Customer Profile Purchases Shipments Claims Orders Clickstreams From Data Mess To Data Products To Instant Value Everywhere CONNECT Data Systems Inventory Replenishment Forecasting … Custom Apps & Microservices Personalization Recommendation Fraud … Confluent Data Streaming Platform A new paradigm for reusable, real-time and trustworthy data products
Why Confluent?
We are the pioneers and experts in data streaming Created by the founders of Confluent “Confluent has an asset that competitors can't replicate: The founders of Confluent were the original developers of Kafka, making it very hard to find anyone that knows the open-source project better than them.”
Confluent Is So Much More Than Apache Kafka Reimagine data streaming everywhere, on-prem and in every major public cloud STREAM Make data in motion self-service, secure, compliant and trustworthy GOVERN Drive greater data reuse with always-on stream processing PROCESS Make it easy to on-ramp and off-ramp data from existing systems and apps CONNECT
Operating Apache Kafka on Your Own Is Difficult I N V E S T M E N T & T I M E V A L U E 4 Early Interest / Experimentation Central Nervous System Mission critical, in Production, but disparate LOBs Identify a Project Mission-critical, connected LOBs As the Kafka footprint expands across more use cases, teams, and environments, it creates significant ongoing operational burden… one that only grows over time, requiring more top engineering talent & resources 5 3 2 1 “One of the challenges with Kafka was its operational complexity, especially as the footprint expanded across our organization. It’s a complex, distributed system, so we had to allocate a lot of our valuable technical resources and expertise to babysit it and keep it running. This also required us to incur additional in-house maintenance costs and risks.” ONEROUS OPERATIONS ● Architecture planning ● Cluster sizing ● Cluster provisioning ● Broker settings ● Zookeeper management EXPENSIVE ENGINEERING, WEAK SECURITY AND AVAILABILITY ● Partition placement & data durability ● Source/sink connectors development & maintenance ● Monitoring & reporting tools setup ● Software patches and upgrades ● Security controls and integrations ● Failover design & planning RISKY GOVERNANCE AND RELIABILITY AT ENTERPRISE SCALE ● Mirroring & geo-replication ● Streaming data governance ● Load rebalancing & monitoring ● Expansion planning & execution ● Utilization optimization & visibility ● Cluster migrations ● Infrastructure & performance upgrades / enhancements INCREASED TOTAL COST OF OWNERSHIP STREAM
Free Up Talent to Move Up the Stack Serverless ● Elastic scaling up & down from 0 to GBps ● Auto capacity mgmt, load balancing, and upgrades High Availability ● 99.99% SLA ● Multi-region / AZ availability across cloud providers ● Patches deployed in Confluent Cloud before Apache Kafka Infinite Storage ● Store data cost- effectively at any scale without growing compute DevOps Automation ● API-driven and/or point-and-click ops ● Service portability & consistency across cloud providers and on-prem Network Flexibility ● Public, VPC, and Private Link ● Seamlessly link across clouds and on-prem with Cluster Linking “Before Confluent Cloud, when we had an outage, developers would have to stop their development work and focus on operations until the problem was sorted out, which in some cases took up to three days. (Now) Confluent takes care of everything for us, so our developers can focus on building new features and applications that add value to the business.” STREAM
$2.57M Total savings Operate 60%+ more efficiently with reduced infrastructure costs, maintenance demands and overhead, and downtime risk 257% 3-year ROI Launch in months rather than years by reducing the burden on your teams with our fully managed cloud service Save on Costs and Increase your ROI Total Economic Impact of using Confluent • Forrester, March 2022 (Link) STREAM “We are in the business of selling and renting clothes. We are not in the business of managing a data streaming platform… If we had to manage everything ourselves, I would’ve had to hire at least 10 more people to keep the systems up and running.” “Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources. With event streaming as a managed service, we had no costly hires to maintain our clusters and no worries about 24x7 reliability."
The Confluent Advantage Create a central nervous system for your data across your enterprise Resilient Infinite Elastic Run Consistently Run Anywhere Stream Governance Stream Processing 120+ Connectors Run Unified Apache Kafka® fully managed, and re-architected to harness the power of the cloud Go way beyond Kafka to build real-time apps quickly, reliably, and securely Flexible deployment options across clouds and on-premises that sync in real-time Cloud Native Complete Everywhere Built and supported by the founders of Apache Kafka® Freedom of Choice Committer-driven Expertise Fully Managed Cloud Service Self-managed Software Training Partners Professional Services Enterprise Support End-to-end Security, Privacy, Compliance and Observability
Why Data Streaming for Data Products and AI
Connect and Share your Data Everywhere “We are making Confluent the true backbone of BHG, including leveraging over 20 Confluent connectors across both modern, cloud-based technologies and legacy systems, to help integrate our critical apps and data systems together.” 37 Cloud Cloud On-premises Run Anywhere Run Unified Run Consistently Supporting any combination of on-prem, hybrid cloud and multicloud deployments for real-time data interoperability STREAM
Unlock Real-time Value From Existing Systems and Applications with Ease Pre-built Connectors Use 120+ self-managed and 70+ fully managed zero-code connectors for instant connectivity from and to existing data systems Custom Connectors Bring your own connector and run it confidently and cost-effectively on our fully managed cloud service Native Integrations Enjoy instant access to streaming data directly within popular platforms, so you never have to leave the tool of your choice “Confluent provides exactly what we dreamed of: an ecosystem of tools to source and sink data from data streams. It’s provided us not only with great data pipeline agility and flexibility but also a highly simplified infrastructure that’s allowed us to reduce costs.” Data Diode CONNECT
Process your data once, process your data right Process and Shape Data As It’s Generated For Greater Reuse 39 Lower costs Improve resource utilization and cost-effectiveness by avoiding redundant processing across silos Data reusability Share consistent and reusable data streams widely with downstream applications and systems Data enrichment Curate, filter, and augment data on-the-fly with additional context to improve completeness and accuracy Real-time everywhere Power low-latency applications and pipelines that react to real-time events, as and when they occur PROCESS
Discover, Understand, and Trust Your Data Streams Where did data come from? Where is it going? Where, when, and how was it transformed? What’s the common taxonomy? What is the current state of the stream? Stream Catalog Increase collaboration and productivity with self-service data discovery Stream Lineage Understand complex data relationships and uncover more insights Stream Quality Deliver trusted, high-quality event streams to the business “Confluent’s Stream Governance suite will play a major role in our expanded use of data in motion and creation of a central nervous system for the enterprise. With the self-service capabilities in stream catalog and stream lineage, we’ll be able to greatly simplify and accelerate the onboarding of new teams working with our most valuable data." GOVERN
Discover Search and understand the data of interest through a self-service user interface Access Request and securely grant access to data streams through simple clicks of buttons Build Start working with your data in real-time to pipe it into existing systems or develop real-time apps Explore and Use Real-time Data Products with a Self-service Portal GOVERN
Without contextualized, trusted, current data… …LLMs can’t drive meaningful value. What is the status of my flight to New York? It is currently delayed by 2 hours and expected to depart at 5 pm GMT. Is there another flight available to the same city that will depart and arrive sooner? What are the seating options and cost? Can your GenAI assistant remember data from an earlier conversation? What is the source of this information? Is this trustworthy? Is it fresh and accurate? How do you augment customer data with real-time data and process them on the fly to provide meaningful insights? The next available flight to New York with United departs later but will arrive faster than your current flight. The only available seats in this flight are first class window seats and costs $1,500.
Confluent enables real-time GenAI-powered applications at scale How? • Integrates disparate operational data across the enterprise in real time for reliable, trustworthy use • Organizes unstructured enterprise data embeddings into vector stores • Decouples customer-facing applications from LLM call management to provide reliable, reactive experiences that scale horizontally • Enables LLMs, vector stores, and embedding models to be treated as modular components that can substituted as technology improves
Enabling Generative AI in the Enterprise Historic Public Data Generative AI Model Intelligent Business-Specific Co-Pilot User Interaction ??
Enabling Generative AI in the Enterprise Historic Public Data Generative AI Model Intelligent Business-Specific Co-Pilot User Interaction DATA STREAMING PLATFORM
Customer Stories and use cases
Data Streaming Use Cases for Media & Entertainment INFRASTRUCTURE USE CASES Streaming Data Pipelines Streaming Applications Event-driven Microservices Real-time AI / ML Hybrid / Multicloud Integration Mainframe Integration Middleware / Messaging Modernization Data Warehouse Pipelining CDC Patterns from Systems of Records Log Aggregation …. …. …. …. and more Shift-left Processing & Governance ANTI-CHEAT, CYBERSECURITY, & FRAUD Cheat detection Bot detection Real time automated account bans DDOS detection and mitigation Suspicious login detection Ad fraud detection USER EXPERIENCE & ENGAGEMENT Cross platform experiences User engagement analysis Live engagement Social media integration Next best action/offer Alerts/Notifications Personalized promotions and loyalty programs Event driven marketing Low-latency user experience BUSINESS OPERATIONS Content finance engineering Budget allocation and marketing attribution Marketing campaign analytics Microtransaction analytics AE to fill out slide if we have an active customer engagement. Blue font indicates existing use of Kafka / Confluent at account.
Challenge: Building a cloud-native, 5G, open RAN (ORAN) network that supports a connectivity platform to enable app communications from any cloud to any device. Solution: Adopting Confluent Cloud to support networking and automation across the hybrid cloud and create consumable data products for developers to leverage. Results: DISH Wireless creates data pipelines using streaming data pipelines built with Confluent Cloud. As a result, its connectivity platform can unlock enterprise Industry 4.0 use cases across industries like retail, manufacturing, and automotive. “We're going up orders of magnitude on the volume of data that we want to transact over Confluent. By having a partner like Confluent, we can start generating value from our network at an unimaginable level today.” — Brian Mengwasser, Vice President & Head of MarketPlace & Apps Design Industry: Telecommunications | Geo: Global | Product: Confluent Cloud Click here to learn more.
Challenge: Decouple a growing number of backend services, manage large amounts of log data, speed time to innovation for future features. Solution: Implemented Apache Kafka® to connect data across the organization and support the move to microservices. Results: ● Reduced time to deployment ● Achieved massive scalability ● Freed developer resources ● Eliminated workflow interruption “We work with Confluent because it is extremely valuable to have access to experts who know the most about Apache Kafka. It’s that feeling of security that has paid off for us.” — Robert Christ, Principal Engineer Industry: Media & Entertainment | Geo: AMER | Product: Confluent Platform
Challenge: Improving time to insights by moving away from conventional batch-based data processing as the sole means of gaining insights and moving toward a stream based approach. Solution: Using streaming data—made possible by Confluent Cloud—to power top-quality, personalized and adaptive experiences for users, inform product roadmap, and drive growth and profitability Results ● Agile decision-making from use of real-time data to monitor and quickly pivot on in-product messaging, campaigns, A/B tests, and experiments ● Adaptive bitrate streaming that ensures seamless playback experience for viewers—without buffering ● Teams can focus on value-added tasks versus managing infrastructure or waiting to access data ● Faster time to market for data products (from days to hours) ● Reduced TCO with Confluent Cloud vs. self-managed Kafka, allowing three FTEs to focus on product development ● Scalability for future use cases, including content delivery optimization and AI/ML “With Confluent, the overarching win is that we’re able to spend more time implementing design and better workflows, making improvements within our current systems, and working on product enhancements. We now have the ability to be driven by business needs.” — Burton Williams, Principal Data Platform Engineer Industry: Video Technology | Geo: AMER | Product: Confluent Cloud Click here to learn more.
Coffee Break
Monoliths to Event-Driven Microservices with Confluent Samuele Dell’Angelo Senior Solution Engineer
54 Tenets for Faster Application Development Challenges with Monoliths and Interservice Communication How We Do It Confluent Cloud in a nutshell Intro to Stream Processing and Apache Flink Agenda 01 02 03 04 05 06
Agenda 55 2. Challenges with Monoliths and Interservice Communication 1. Tenets for Faster Application Development 3. How We Do It 4. Confluent Cloud in a nutshell 5. Intro to Stream Processing and Apache Flink
Tenets for Faster Application Development 56 Eliminate Dependencies Reduce Technical Debt and TCO Build and Integrate Scalable Apps
What’s Holding You Back? 57
58 Challenges with Monoliths … Monolith ● Slow: Smallest change requires end-end testing ● Unreliable: A simple error can bring down the entire app ● Expensive: Changes to tech stack are expensive. Barrier to adopting new programming frameworks Services Database
59 Microservices ● Faster release cycles: Smaller modules that can Independently deployed and upgraded ● Improved Scalability and Reliability: Independently scale services and recover from errors faster ● Developer autonomy: Choose the right programming language and data store for each service Services Databases Services Database … Have Led to the Adoption of Microservices Monolith ● Slow: Smallest change requires end-end testing ● Unreliable: A simple error can bring down the entire app ● Expensive: Changes to tech stack are expensive. Barrier to adopting new programming frameworks
Two Approaches For Interservice Communication 60 REST APIs Synchronous request/response communication Messaging Queues Synchronous or asynchronous communication
Service 4 Service 3 Challenges with Using REST APIs for Interservice Communication 61 REST APIs Synchronous request/response Challenges ● Slow Tight coupling means business logic is rewritten to add new services. ● Unreliable Requires costly linear scaling. Unexpected errors are difficult to recover from. ● Expensive High operational overhead of managing services, at scale. Service 2 Service 5 Service 6 Service 1 API API API API API API
Challenges with Using Message Queues for Interservice Communication 62 Messaging Queues Synchronous or asynchronous communication Challenges ● Slow Lack a common structure, leads to inter-team dependencies. ● Unreliable Cannot scale dynamically. Ephemeral persistence. ● Expensive Cannot easily integrate with cloud-native apps. Service 2 Service 5 Service 4 Service 3 Service 6 Service 1 Messaging Queues (No common structure to share data, lacks built-in stream processing, ephemeral message persistence, low fault-tolerance)
MicroServices -> Event Driven Microservices 63
Why does it make sense? Microservices Distributed Semi-loose with API versioning Requires a consistent distributed system Synchronous Event Driven Microservices Central, synchronized through events Completely decoupled (Fire and forget) The streaming platform reduces the dependencies to external system Reactive (asynchronous) State Coupling E2E Testing Execution Model 64
How We Do It: 65
How to Evolve into Event-Driven Microservices 66 CDC off legacy database / mainframe Leverage Confluent’s connector ecosystem to easily CDC data off legacy mainframes and databases Modernize legacy MQ to event streaming platform Decouple MQ from your application by sending data to Confluent Enrich events with less code using stream processing Join, summarize, and filter event streams in real time, reducing the number of microservices needed Ensure data compatibility with Schema Validation Validate data compatibility and enable evolution of schemas while preserving backwards compatibility 2 3 1 4 Mainframe Microservice 1 Microservice 2 CDC Connector Legacy MQ MQ Connector DB KStreams / KsqlDB / Flink Incompatible Data Schema Registry Validated & Compatible Microservice 1 Microservice 2 Microservice 2 Microservice 1 Microservice 2 Microservice
Today’s Pattern Write to My Database Then Use CDC (Write Through) Write to Kafka Then Sink to My Database Commit the Horror That is Dual Writes
6 9 Suggested approach: Outbox + CDC Patterns In DDD systems an Aggregate’s internal representation of data may/should be different from the outside contract, In that scenario, outbox pattern can be applied. Application writes data to its table in the database and also to an outbox table within a single Transaction. The outbox table is sourced to Kafka topics. (Write Through)
7 0 Replace XA transactions for multiple database writes ● One Application needs to write to two different database storage engines where XA transactions are not supported. ● Instead of XA transaction, the Application writes data to a single database table, which is sourced to a Kafka topic by a Source connector. Multiple Sink connectors can be used to sink data to different storage engines. Suggested approach: Outbox + CDC Patterns
Transactional outbox with Confluent 71 Outbox Orders Confluent Platform Connect KsqlDB Ksrteam Relay (no code) Stream processing (code) Applications Stream processing (SQL) Event Broker Connect Other Services
7 2 Other approach: Database write aside In its default form, this pattern guarantees dual-write for most use cases. However, should the database transaction fail at commit time (say, because the database server has crashed) the write to Kafka cannot be rolled back unless transactions have been enabled. For many use cases, this eventuality will be tolerable as the dual-write can be retried once the failure is fixed, and most event consumers will implement idempotence anyway. However, application programmers need to be aware that there is no firm guarantee. Transactional messaging systems like Kafka can be used to provide stronger guarantees so long as all event consumers have the transactions feature enabled. (Database write aside) Write to a database, then write to Kafka. Perform the write to Kafka as the last step in a database transaction to ensure an atomic dual commit (aborting the transaction if the write to Kafka fails).
7 3 Temptation: Dual write ● Database transactions and Kafka transactions are separate ● To perform them atomically would need to be done as a distributed transaction ● You can use Spring Transaction Manager ○ Chaining Kafka and Database Transactions with Spring Boot: An In-Depth Look ○ Examples of Kafka Transactions with Other Transaction Managers (Dual write) Write to a database and to Kafka using an atomic approach. KAFKA DATABASES Write APP Write
74 Are you safe with this approach?
75 Kafka at this moment doesn’t support 2 Phase Commit
76 Next Step
KIP-939: Support Participation in Two Phase Commit Enables consistent and atomic update of Kafka and other transaction stores 3. On the Horizon PRODUCER APP PRODUCER APP COMMIT COMMIT PREPARED ABORT ABORT
Solving 2PC with KIP-939 Red Flags InitProducerId must be called on client restarts, it currently aborts any ongoing transactions Transaction.max.timeout.ms can abort transactions independent of the transaction coordinator
Example Flow 7 9 KAFKA DATABASES 2 5 2 3 1 APP 4 1. Commit or Abort Previous Transaction in Kafka (if any) 2. Start New Transactions in Kafka and the Database; Write App Data 3. 2PC Prepare Phase: Prepare Kafka Transactions 4. Persist 2PC Decision 5. 2PC Commit Phase: Commit Kafka Transactions
FLIP-319 – Integrating with Kafka 2PC Transactions Hardens EOS Across Flink and Kafka Simplifies Upgrading Kafka Clients Used by Flink
8 1 KAFKA DATABASES SOURCES SINKS FILES KAFKA APPS DATABASES
82
An Example
E-Commerce Application (Monolith) 84 Catalog Order Payment On-Prem Monolith (E-commerce App) Load Balancer Client/Web Application Writes Reads Writes Reads
Step 1: Route Reads to Catalog Microservice 85 Catalog Order Payment On-Prem Monolith (E-commerce App) Load Balancer Client/Web Application Writes to Catalog module Other Writes and Reads Writes Reads Writes CDC Source Connector Streams (Reads) Reads from Microservice Partially replaces Catalog Module Cloud Native Catalog Microservice
Step 2: Fully Replace Catalog Module 86 Order Payment On-Prem Monolith (E-commerce App) Load Balancer Client/Web Application Other Writes and Reads Writes Reads Writes CDC Source +Sink Connector Cloud Native Catalog Microservice Streams (Reads) Reads from Microservice Reads Streams (Writes) Writes to Microservice Third Party Consumer Apps
Challenge: eBay Korea wanted to enable continued growth across multiple e-commerce platforms that had been in place for more than 15 years. Solution: eBay Korea leverages Confluent to help migrate from a monolithic architecture to a microservices architecture that enables more independence, greater flexibility, and improved scalability. Results: ● Simplified maintenance and operations for improved productivity ● Stable, high-throughput streaming ● Rapid support ● Faster time to market “Although we had some experience running open source Kafka ourselves, we recognized that we needed a trusted partner to help us resolve issues that we’d encounter with a larger-scale deployment. Confluent not only had a local office to provide that support, they also had the best known, verified solution, which was what we wanted for our mission-critical e-commerce platforms.” — Hudson Lee, Leader of Platform Engineering
Confluent Cloud in a Nutshell
The Confluent Product Advantage 89 Re-imagined Kafka experience for the Cloud Be everywhere our customers want to be COMPLETE EVERYWHERE CLOUD NATIVE Enable developers to reliably & securely build next-gen apps faster
90 Cloud-native data streaming platform built by the founders of Apache Kafka® KORA: THE KAFKA ENGINE FOR THE CLOUD STREAM Everywhere Be everywhere our customers want to be Cloud-native Re-imagined Kafka experience for the Cloud Complete Go above and beyond Kafka with a complete data streaming platform CONNECT GOVERN PROCESS Confluent Data Streaming Platform
Kora Architecture We spent 5 million+ hours to re-invent Kafka into a truly cloud-native engine NETWORK COMPUTE OBJECT STORAGE CUSTOMERS AZ AZ AZ Cells Cells Cells Multi-Cloud Networking & Routing Tier Metadata Durability Audits Data Balancing Health Checks Real-time feedback data GLOBAL CONTROL PLANE S E C U R I T Y
THE KAFKA ENGINE FOR THE CLOUD STREAM RESILIENT COST EFFECTIVE 10X FAST ELASTIC 16X 30X
Using fully managed connectors is the fastest, most efficient way to break data silos Self-managed connector Accelerated time-to-value • Increased developer productivity • Reduced operational burden ● Pre-built but requires manual installation / config efforts to set-up and deploy connectors ● Perpetual management and maintenance of connectors that leads to ongoing tech debt ● Risk of downtime and business disruption due to connector / Connect cluster related issues Fully managed connector Custom-built connector ● Streamlined configurations and on-demand provisioning of your connectors ● Eliminates operational overhead and management complexity with seamless scaling and load balancing ● Reduced risk of downtime with Confluent Cloud’s 99.95% SLA for all your mission critical use cases ● Costly to allocate resources to design, build, test, and maintain non-differentiated data integration components ● Delays time-to-value, taking up to 3-6+ engineering months to develop ● Perpetual management and maintenance increases tech debt and risk of downtime
Connect your entire business with just a few clicks 70+ fully managed connectors Amazon S3 Amazon Redshift Amazon DynamoDB Google Cloud Spanner AWS Lambda Amazon SQS Amazon Kinesis Azure Service Bus Azure Event Hubs Azure Synapse Analytics Azure Blob Storage Azure Functions Azure Data Lake Google BigTable
Stream Governance is provided fully managed on Confluent Cloud Stream Quality Deliver trusted, high-quality data streams to the business Stream Catalog Increase collaboration and productivity with self-service data discovery Stream Lineage Understand complex data relationships and uncover more insights (Schema Registry & Validation) Discover, understand, and trust your data streams with the industry’s only fully managed governance suite for Apache Kafka® Stream Governance
schema registry SCHEMA METADATA SCHEMA METADATA Producer App 3 ... Producer App 1 Producer App 2 Consumer App 3 ... Consumer App 1 Consumer App 2 UNIVERSAL LANGUAGE & QUALITY CONTROL Stream Quality Deliver trusted, high-quality data streams to the business Avro, Protocol Buffers, and JSON serialization formats supported validate schemas before producing validate schemas before consuming Schema Registry is available with a 99.95% Uptime SLA Maintain trusted, compatible data streams across cloud and hybrid environments with shared schemas that sync in real time, compatibility rules, and schema references plugin
Intro to Stream Processing and Apache Flink
Real-time processing Power low-latency applications and pipelines that react to real-time events and provide timely insights Data reusability Share consistent and reusable data streams widely with downstream applications and systems Data enrichment Curate, filter, and augment data on-the-fly with additional context to improve completeness, accuracy, & compliance Efficiency Improve resource utilization and cost-effectiveness by avoiding redundant processing across silos “With Confluent’s fully managed Flink offering, we can access, aggregate, and enrich data from IoT sensors, smart cameras, and Wi-Fi analytics, to swiftly take action on potential threats in real time, such as intrusion detection. This enables us to process sensor data as soon as the events occur, allowing for faster detection and response to security incidents without any added operational burden.” Effortlessly filter, join, and enrich your data streams with Apache Flink
Blob storage 3rd party app Databases Data Warehouse Database SaaS app Low latency apps and data pipelines Consistent, reusable data products Optimized resource utilization Process data streams in-flight to maximize actionability, fidelity, and portability
Pre-Packaged stream processing workloads such as: ● deduplication topic ● masking fields EVENT-DRIVEN APPLICATIONS Flink Actions
C price>lag(price) D price<lag(price) C price>lag(price) B price<lag(price) A Double Bottom Period & Volume Price Develop applications using fine-grained control over how time progresses and data is grouped together using: ● Hopping, tumbling, session windows ● OVER aggregations ● Pattern matching with MATCH_RECOGNIZE EVENT-DRIVEN APPLICATIONS Recognize patterns and react to events in a timely manner
Get up-to-date results to power dashboards or applications requiring continuous updates using: ● Materialized views ● Temporal analytic functions ● Interactive queries Account Balance A $15 B $2 C $15 Account A, +$10 Account B, +$12 Account C, +$5 Account B, -$10 Account C, +$10 Account A, -$5 Account A, +$10 Time REAL-TIME ANALYTICS Analyze real-time data streams to generate important business insights
Create new enriched and curated streams of higher value using: ● Data transformations ● Streaming joins, temporal joins, lookup joins, and versioned joins ● Fan out queries, multi-cluster queries 103 t1, 21.5 USD t3, 55 EUR t5, 35.3 EUR t0, EUR:USD=1.00 t2, EUR:USD=1.05 t4: EUR:USD=1.10 t1, 21.5 USD t3, 57.75 USD t5, 38.83 USD Currency rate Orders STREAMING DATA PIPELINES Build streaming data pipelines to inform real-time decision making
INSERT INTO enriched_reviews SELECT id , review , invoke_openai(prompt,review) as score FROM product_reviews ; K N B Kate 4 hours ago This was the worst decision ever. Nikola 1 day ago Not bad. Could have been cheaper. Brian 3 days ago Amazing! Game Changer! K N B Kate ★★★★★ 4 hours ago This was the worst decision ever. Nikola ★★★★★ 1 day ago Not bad. Could have been cheaper. Brian ★★★★★ 3 days ago Amazing! Game Changer! The Prompt “Score the following text on a scale of 1 and 5 where 1 is negative and 5 is positive returning only the number” DATA STREAMING PLATFORM Enrich real-time data streams with Generative AI directly from Flink SQL Next
Grazie!

More Related Content

PPTX
Azure integration services from the IT Professional perspective
byAlessandro Moura
 
PPTX
Altis: AWS Snowflake Practice
byAltis Consulting
 
PDF
Mdm: why, when, how
byJean-Michel Franco
 
PDF
Building an Effective Data & Analytics Operating Model A Data Modernization G...
byMark Hewitt
 
PPTX
Microsoft Azure Data Factory Hands-On Lab Overview Slides
byMark Kromer
 
PDF
DMBOK and Data Governance
byPeter Vennel PMP,SCEA,CBIP,CDMP
 
PDF
Modern Agile - Joshua Kerievsky
byAgileSparks
 
PDF
The Agile Adoption Roadmap (Keynote by Tim Abbott)
byAgile Days Middle East
 
Azure integration services from the IT Professional perspective
byAlessandro Moura
 
Altis: AWS Snowflake Practice
byAltis Consulting
 
Mdm: why, when, how
byJean-Michel Franco
 
Building an Effective Data & Analytics Operating Model A Data Modernization G...
byMark Hewitt
 
Microsoft Azure Data Factory Hands-On Lab Overview Slides
byMark Kromer
 
DMBOK and Data Governance
byPeter Vennel PMP,SCEA,CBIP,CDMP
 
Modern Agile - Joshua Kerievsky
byAgileSparks
 
The Agile Adoption Roadmap (Keynote by Tim Abbott)
byAgile Days Middle East
 

What's hot

PPTX
Integrate with coupa
byCoupa Software
 
PPTX
Data Lakehouse, Data Mesh, and Data Fabric (r1)
byJames Serra
 
PDF
Digital Core Transformation - SAP S/4HANA
byDeloitte Switzerland
 
PDF
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
byDATAVERSITY
 
PPT
SAP BusinessObject's Webi Rich Client
byEric Molner
 
PDF
Event Driven Architecture
byLourens Naudé
 
PDF
DAS Slides: Building a Data Strategy — Practical Steps for Aligning with Busi...
byDATAVERSITY
 
PPTX
Migrating Data and Databases to Azure
byKaren Lopez
 
PDF
Data Modeling Fundamentals
byDATAVERSITY
 
PPTX
Dynamics 365 for Finance and Operations - Power BI
byJuan Fabian
 
PDF
Free Online Agile & SCRUM Study Training Material for PMI ACP Certification P...
byGlobalSkillup
 
PDF
Azure governance v4.0
byMarcos Oikawa
 
PDF
Cloud Data Warehouses
byAsis Mohanty
 
PPT
Doing Agile Isnt The Same As Being Agile
bylazygolfer
 
PDF
Business Value Metrics for Data Governance
byDATAVERSITY
 
PPTX
Data mesh
byManojKumarR41
 
PDF
Scrum 101: Introduction to Scrum
byArrielle Mali
 
PDF
Coaching Scrum Teams
bybmyllerup
 
PDF
Modeling the Backstory with the ArchiMate Motivation Extension
byIver Band
 
PDF
Pi Planning with Easy Agile Programs for Jira
bySean Blake
 
Integrate with coupa
byCoupa Software
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
byJames Serra
 
Digital Core Transformation - SAP S/4HANA
byDeloitte Switzerland
 
Data Architecture, Solution Architecture, Platform Architecture — What’s the ...
byDATAVERSITY
 
SAP BusinessObject's Webi Rich Client
byEric Molner
 
Event Driven Architecture
byLourens Naudé
 
DAS Slides: Building a Data Strategy — Practical Steps for Aligning with Busi...
byDATAVERSITY
 
Migrating Data and Databases to Azure
byKaren Lopez
 
Data Modeling Fundamentals
byDATAVERSITY
 
Dynamics 365 for Finance and Operations - Power BI
byJuan Fabian
 
Free Online Agile & SCRUM Study Training Material for PMI ACP Certification P...
byGlobalSkillup
 
Azure governance v4.0
byMarcos Oikawa
 
Cloud Data Warehouses
byAsis Mohanty
 
Doing Agile Isnt The Same As Being Agile
bylazygolfer
 
Business Value Metrics for Data Governance
byDATAVERSITY
 
Data mesh
byManojKumarR41
 
Scrum 101: Introduction to Scrum
byArrielle Mali
 
Coaching Scrum Teams
bybmyllerup
 
Modeling the Backstory with the ArchiMate Motivation Extension
byIver Band
 
Pi Planning with Easy Agile Programs for Jira
bySean Blake
 

Similar to Unlocking value with event-driven architecture by Confluent

PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
PDF
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
byKai Wähner
 
PDF
Real-time processing of large amounts of data
byconfluent
 
PDF
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
byconfluent
 
PDF
Confluent Partner Tech Talk with BearingPoint
byconfluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
PPTX
Building Serverless EDA w_ AWS Lambda (1).pptx
byAhmed791434
 
PDF
Citi Tech Talk: Event Driven Kafka Microservices
byconfluent
 
PDF
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
byconfluent
 
PPTX
Digital Transformation Mindset - More Than Just Technology
byconfluent
 
PDF
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
byconfluent
 
PDF
Confluent & GSI Webinars series: Session 2
byconfluent
 
PDF
Apache kafka event_streaming___kai_waehner
byconfluent
 
PDF
APAC Exec Roundtable
byconfluent
 
PDF
Message Driven and Event Sourcing
byPaolo Castagna
 
PDF
Serverless and Streaming: Building ‘eBay’ by ‘Turning the Database Inside Out’
byconfluent
 
PDF
Processing Real-Time Data at Scale: A streaming platform as a central nervous...
byconfluent
 
PDF
Pivoting event streaming, from PROJECTS to a PLATFORM
byconfluent
 
PDF
ANZ C-Level Roundtable
byconfluent
 
PDF
[INFOGRAPHIC] Event-driven Business: How to Handle the Flow of Event Data
byconfluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
byconfluent
 
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
byKai Wähner
 
Real-time processing of large amounts of data
byconfluent
 
Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...
byconfluent
 
Confluent Partner Tech Talk with BearingPoint
byconfluent
 
Five Things You Need to Know About Data Streaming in 2025
byconfluent
 
Building Serverless EDA w_ AWS Lambda (1).pptx
byAhmed791434
 
Citi Tech Talk: Event Driven Kafka Microservices
byconfluent
 
Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud...
byconfluent
 
Digital Transformation Mindset - More Than Just Technology
byconfluent
 
Jay Kreps | Kafka Summit NYC 2019 Keynote (Events Everywhere) | CEO, Confluent
byconfluent
 
Confluent & GSI Webinars series: Session 2
byconfluent
 
Apache kafka event_streaming___kai_waehner
byconfluent
 
APAC Exec Roundtable
byconfluent
 
Message Driven and Event Sourcing
byPaolo Castagna
 
Serverless and Streaming: Building ‘eBay’ by ‘Turning the Database Inside Out’
byconfluent
 
Processing Real-Time Data at Scale: A streaming platform as a central nervous...
byconfluent
 
Pivoting event streaming, from PROJECTS to a PLATFORM
byconfluent
 
ANZ C-Level Roundtable
byconfluent
 
[INFOGRAPHIC] Event-driven Business: How to Handle the Flow of Event Data
byconfluent
 

More from confluent

PDF
Santander Stream Processing with Apache Flink
byconfluent
 
PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
PDF
Migration, backup and restore made easy using Kannika
byconfluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
PDF
Unlocking the Power of IoT: A comprehensive approach to real-time insights
byconfluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
PPTX
Workshop híbrido: Stream Processing con Flink
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
PDF
Building API data products on top of your real-time data infrastructure
byconfluent
 
Santander Stream Processing with Apache Flink
byconfluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
byconfluent
 
Migration, backup and restore made easy using Kannika
byconfluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
byconfluent
 
Unlocking the Power of IoT: A comprehensive approach to real-time insights
byconfluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
byconfluent
 
Speed Wins: From Kafka to APIs in Minutes
byconfluent
 
Workshop híbrido: Stream Processing con Flink
byconfluent
 
Data in Motion Tour Seoul 2024 - Keynote
byconfluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
byconfluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
byconfluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
byconfluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
byconfluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
byconfluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
byconfluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
byconfluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
byconfluent
 
Il Data Streaming per un’AI real-time di nuova generazione
byconfluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
byconfluent
 
Building API data products on top of your real-time data infrastructure
byconfluent
 

Recently uploaded

PPTX
Key Benefits of Odoo Customization Services
byCandidRoot Solutions Private Limited
 
PDF
AI/ML Infra Meetup | Open Source Michelangelo: Uber's Predictive to Generativ...
byAlluxio, Inc.
 
PDF
2025_11_19 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
PDF
Connecting Strategy to Delivery Teams Using OnePlan.pdf
byOnePlan Solutions
 
PPTX
Harnessing BUSY Software for Smart Business Management.pptx
byMoving Cloud
 
PDF
AI/ML Infra Meetup | Unlock the Future of Generative AI: TorchTitan's Latest ...
byAlluxio, Inc.
 
DOCX
Open Source BI Tableau Alternatives You Can Trust.docx
byVarsha Nayak
 
PDF
How Integrating Copilot and AI Agents in Microsoft Dynamics 365
byTechWize
 
PPTX
Performance Testing Transformation - LTB 2024
byAlexander Podelko
 
PDF
A new Vision of Software Sustainability and its Engineering: Reflections and ...
byPatricia Lago
 
PPTX
Detecting Bad Apples: Understanding macOS Malware TTPs by Surya Teja
byCysinfo Cyber Security Community
 
PDF
Speculative Automated Refactoring of Imperative Deep Learning Programs to Gra...
byRaffi Khatchadourian
 
PDF
How to Make Your AI Reliable: Comprehensive Guide to Testing AI Applications ...
byQATestLab Quality is a rule
 
PPTX
Developing a Language Plugin LSP versus the Joy of Learning
byslaleksandr28
 
PPTX
Guard Tour Patrol System in Singapore_ Real-Time Tracking and Free Payroll.pptx
byjaharikam
 
PDF
6 Hotel Booking Trends You Can’t Ignore in 2025.pdf
byHotelogix
 
PDF
Quantum TechDocs - Technology & Compliance.pdf
byietmsoftware1
 
PDF
Threat Hunting with Garuda: From Manual Analysis to AI-Driven Hunting By Monn...
byCysinfo Cyber Security Community
 
PPTX
Working of Minimax Algorithm with Alpha-Beta Pruning.pptx
byMuhammadHassanArshad2
 
PPTX
Short Film Script Development Process HARMONIA
byzdolezalmedia
 
Key Benefits of Odoo Customization Services
byCandidRoot Solutions Private Limited
 
AI/ML Infra Meetup | Open Source Michelangelo: Uber's Predictive to Generativ...
byAlluxio, Inc.
 
2025_11_19 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
Connecting Strategy to Delivery Teams Using OnePlan.pdf
byOnePlan Solutions
 
Harnessing BUSY Software for Smart Business Management.pptx
byMoving Cloud
 
AI/ML Infra Meetup | Unlock the Future of Generative AI: TorchTitan's Latest ...
byAlluxio, Inc.
 
Open Source BI Tableau Alternatives You Can Trust.docx
byVarsha Nayak
 
How Integrating Copilot and AI Agents in Microsoft Dynamics 365
byTechWize
 
Performance Testing Transformation - LTB 2024
byAlexander Podelko
 
A new Vision of Software Sustainability and its Engineering: Reflections and ...
byPatricia Lago
 
Detecting Bad Apples: Understanding macOS Malware TTPs by Surya Teja
byCysinfo Cyber Security Community
 
Speculative Automated Refactoring of Imperative Deep Learning Programs to Gra...
byRaffi Khatchadourian
 
How to Make Your AI Reliable: Comprehensive Guide to Testing AI Applications ...
byQATestLab Quality is a rule
 
Developing a Language Plugin LSP versus the Joy of Learning
byslaleksandr28
 
Guard Tour Patrol System in Singapore_ Real-Time Tracking and Free Payroll.pptx
byjaharikam
 
6 Hotel Booking Trends You Can’t Ignore in 2025.pdf
byHotelogix
 
Quantum TechDocs - Technology & Compliance.pdf
byietmsoftware1
 
Threat Hunting with Garuda: From Manual Analysis to AI-Driven Hunting By Monn...
byCysinfo Cyber Security Community
 
Working of Minimax Algorithm with Alpha-Beta Pruning.pptx
byMuhammadHassanArshad2
 
Short Film Script Development Process HARMONIA
byzdolezalmedia
 

Unlocking value with event-driven architecture by Confluent

  • 1.
    Unlocking Value WithEvent-Driven Architecture By Confluent 1
  • 2.
    9:30 - 10:00 10:00- 10:15 10:15 - 11:15 11:15 - 11:30 11:30 - 12:30 12.30 - 13:00 Welcome Coffee SKY verso l’Event-Driven Architecture Daniele De Angelis, Senior Specialist – Architecture - Sky Italia Luca Cogliandro, Senior Manager – Software Engineering Integration - Sky Italia Sessione 1: How Event-Driven Architectures Enable Business Value Samuele Dell'Angelo, Senior Solutions Engineer - Confluent Break Sessione 2: Monoliths to Event-Driven Microservices with Confluent Samuele Dell'Angelo, Senior Solutions Engineer - Confluent Networking Light Lunch Agenda
  • 3.
    SKY verso l’Event-DrivenArchitecture Daniele De Angelis Senior Specialist – Architecture,Sky Italia Luca Cogliandro Senior Manager – Software Engineering Integration, Sky Italia 3
  • 4.
    API Exposure Adapter GET Composite Service Asyncrhonous Dispositive Commonservices Proactive Monitoring (shift-left) Adapter POST Fast Information Cloud/Paas Infrastructure Customer Touchpoint (LightHouse) Event Driven Architecture STEP 1 Data Driven STEP 3 STEP 2 Guidelines REST Api Design Pass Through Design Business Logic Orchestration Choreography and Event sourcing Application Design Error Handling and Resubmission Logging and monitoring Pass Through GET & POST (synchronous API) Logging Every Hermes Enterprise Framework pillar is based on a set of Enterprise and Integration Guidelines. These pillars translate into a set of Design Patterns to be used within the framework design and implementation process CI/CD pipelines
  • 5.
    Event-driven architecture isdefined as a software design pattern in which decoupled applications can asynchronously publish and subscribe to events via an event broker Event messages are published to a message broker, and interested subscribers or services can consume these events Participants exchange events without a centralized point of control. Each local transaction publishes domain events that trigger local transactions in other services. Benefits ➢ Loose coupling ➢ Fault tolerance ➢ Cost-effective Challenges ➢ Ordering ➢ Idempotency ✔ Consents ✔ RTD KPIs Benefits ➢ Distributed transaction ➢ Rollback operation ➢ Local transactions isolation Challanges ➢ Complexity ➢ Resilient implementation ➢ Observability ✔ Order creation Message broker Service A (Local transaction) Service B (Local transaction) Service C (Local transaction) Msg Input A Msg Input B Msg Input C Msg Output A Msg Output B Msg Output C
  • 6.
    Event-driven Architecture basedon Kafka Confluent Cloud • Platform PaaS • IAC CICD Automation • Production Cluster Multi Zone • Test Cluster Single Zone Confluent Platform Data Platform RTD • Topic Name Strategy 🡪 Business Domain • Schema Registry 🡪 Data Governance Producer Consumer Service Exposure • KPI ✔ 4 Million events produced and consumed in 10 minutes ✔ Live update major KPI Tracking Tools Tibco Data Platform Consents KPI events events AWS • Infrastructural • Consumer Lag • DLT not empty • No events into Topic
  • 7.
    How Event-Driven Architectures EnableBusiness Value Samuele Dell’Angelo Senior Solution Engineer, Confluent 7
  • 8.
    01 02 03 04 05 06 Intro to EventStreaming: What is Event Streaming? Data: The New Currency of Business Why you need Data Streaming Why Confluent Why Data Streaming for Data Products and AI Customer Stories and use cases Agenda
  • 9.
    Intro to EventStreaming
  • 10.
    Software is Changingevery Aspect of life 10
  • 11.
  • 12.
    What Are Events Transportation TPMSsensor in Carol’s car detected low tire-pressure at 5:11am. Kafka Banking Alice sent $250 to Bob on Friday at 7:34pm. Kafka Retail Sabine’s order of a Fujifilm camera was shipped at 9:10am. Kafka
  • 13.
    The Shift inThinking: Event Streams 13 Rich User experiences Real-time events Real-time Event Streams A Sale A shipment A Trade Customer Actions Data driven operation Real-time Analytics Microservices Stream Processing
  • 14.
    Data: The NewCurrency of Business Success
  • 15.
    Data is theFuel for a Modern Business Loyalty Rewards Payments Processing Real-time Trades Fraud Detection Personalized Recommendations
  • 16.
    Unlocking Data andAnalytics is Much Harder than it Should Be
  • 17.
    17 There are TwoMain Locations for Data OPERATIONAL ESTATE Serve the needs of applications to transact with customers in real-time ANALYTICAL ESTATE Support after-the-fact business analysis and reporting for various stakeholders
  • 18.
    OPERATIONAL ESTATE Serve theneeds of applications to transact with customers in real-time ANALYTICAL ESTATE Support after-the-fact business analysis and reporting for various stakeholders 18 As Data Needs Multiplied, Data Complexity Escalated
  • 19.
    … creating thisgiant mess!
  • 20.
    Your teams arebuilding applications on this messy data architecture Recommendations Real-time payments Branch monitoring Real-time bot detection Real-time alerts Live customer chatbot Omni-channel orders Real-time inventory Mobile banking Fraud detection
  • 21.
    Why you needData Streaming
  • 22.
    Data Streaming: ANew Paradigm to Set Data in Motion Continuously ingest, process, govern and share streams of data in real-time Real-time Data A Transfer A Login A Trade A Customer Experience Rich Front-End Customer Experiences Real-Time Backend Operations New Products and Services “We need to shift our thinking from everything at rest to everything in motion.”
  • 23.
    From Giant Messto Central Nervous System DATA STREAMING PLATFORM “For a typical Fortune 1000 company, just a 10% increase in data accessibility will result in more than $65 million additional net income.”
  • 24.
    Shift Your Thinking Point-in-timequeries on static data Continuous processing at generation time Tightly coupled point-to-point integrations Multi-subscriber decoupled architecture Producer-presented, governed & reusable data Complex data wrangling downstream Batch-based data integration Event-driven continuous data flows
  • 25.
    Apply a DataProduct Thinking FINANCIAL SERVICES RETAIL & E-COMMERCE MANUFACTURING HEALTHCARE TRANSPORTATION GAMING Customers Accounts Web logins Clickstream Transactions Payments Branches Claims Trades Customers Accounts Purchases Web clickstream Inventory Stores Employees Shipments Machine Telemetry Inventory Orders Shipments Suppliers Assets Equipment Patients Accounts Diagnoses Treatments Claims Readings Providers Players Achievements Profiles Games Purchases Vehicle Telemetry Drivers Customers Conditions Deliveries Repairs Insurers Chats High Scores Data products: Real-time, discoverable, contextualized, trustworthy and reusable
  • 26.
    Use Data Productsto Build New Business Applications Companies that treat data like a product can: ● Reduce the time it takes to implement new use cases by as much as 90% ● Decrease total ownership costs by up to 30% (technology, development, and maintenance) ● Reduce their risk and data governance burden Customers Accounts Web Logins Transactions Fraud Detection Customers Accounts Inventory Purchases Web Clickstream Customer Loyalty Vehicle Telemetry Repairs Drivers Customers Delivery Management Data Products Use Cases
  • 27.
    Build an EffectiveData Flywheel Strategy by Reusing Data Products Across Multiple Use Cases Supply chain systems Watch lists Profile mgmt Incident mgmt Customers Threat vector Customer churn Transactions Payments Mainframe data Inventory Weather Telemetry IoT data Notification engine Payroll systems Distribution Order fulfillment CRM systems Mobile application Personalization Clickstreams Customer loyalty Change logs Purchases Recommendation engine Web application Payment service Heatmap service ITSM systems Central log systems (CLS) Fraud & SIEM systems Alerting systems AI/ML engines Visualization apps
  • 28.
    PROCESS GOVERN CONNECT STREAM Accounts Customer Profile Purchases Shipments Claims Orders Clickstreams From DataMess To Data Products To Instant Value Everywhere CONNECT Data Systems Inventory Replenishment Forecasting … Custom Apps & Microservices Personalization Recommendation Fraud … Confluent Data Streaming Platform A new paradigm for reusable, real-time and trustworthy data products
  • 29.
  • 30.
    We are thepioneers and experts in data streaming Created by the founders of Confluent “Confluent has an asset that competitors can't replicate: The founders of Confluent were the original developers of Kafka, making it very hard to find anyone that knows the open-source project better than them.”
  • 31.
    Confluent Is SoMuch More Than Apache Kafka Reimagine data streaming everywhere, on-prem and in every major public cloud STREAM Make data in motion self-service, secure, compliant and trustworthy GOVERN Drive greater data reuse with always-on stream processing PROCESS Make it easy to on-ramp and off-ramp data from existing systems and apps CONNECT
  • 32.
    Operating Apache Kafkaon Your Own Is Difficult I N V E S T M E N T & T I M E V A L U E 4 Early Interest / Experimentation Central Nervous System Mission critical, in Production, but disparate LOBs Identify a Project Mission-critical, connected LOBs As the Kafka footprint expands across more use cases, teams, and environments, it creates significant ongoing operational burden… one that only grows over time, requiring more top engineering talent & resources 5 3 2 1 “One of the challenges with Kafka was its operational complexity, especially as the footprint expanded across our organization. It’s a complex, distributed system, so we had to allocate a lot of our valuable technical resources and expertise to babysit it and keep it running. This also required us to incur additional in-house maintenance costs and risks.” ONEROUS OPERATIONS ● Architecture planning ● Cluster sizing ● Cluster provisioning ● Broker settings ● Zookeeper management EXPENSIVE ENGINEERING, WEAK SECURITY AND AVAILABILITY ● Partition placement & data durability ● Source/sink connectors development & maintenance ● Monitoring & reporting tools setup ● Software patches and upgrades ● Security controls and integrations ● Failover design & planning RISKY GOVERNANCE AND RELIABILITY AT ENTERPRISE SCALE ● Mirroring & geo-replication ● Streaming data governance ● Load rebalancing & monitoring ● Expansion planning & execution ● Utilization optimization & visibility ● Cluster migrations ● Infrastructure & performance upgrades / enhancements INCREASED TOTAL COST OF OWNERSHIP STREAM
  • 33.
    Free Up Talentto Move Up the Stack Serverless ● Elastic scaling up & down from 0 to GBps ● Auto capacity mgmt, load balancing, and upgrades High Availability ● 99.99% SLA ● Multi-region / AZ availability across cloud providers ● Patches deployed in Confluent Cloud before Apache Kafka Infinite Storage ● Store data cost- effectively at any scale without growing compute DevOps Automation ● API-driven and/or point-and-click ops ● Service portability & consistency across cloud providers and on-prem Network Flexibility ● Public, VPC, and Private Link ● Seamlessly link across clouds and on-prem with Cluster Linking “Before Confluent Cloud, when we had an outage, developers would have to stop their development work and focus on operations until the problem was sorted out, which in some cases took up to three days. (Now) Confluent takes care of everything for us, so our developers can focus on building new features and applications that add value to the business.” STREAM
  • 34.
    $2.57M Total savings Operate 60%+more efficiently with reduced infrastructure costs, maintenance demands and overhead, and downtime risk 257% 3-year ROI Launch in months rather than years by reducing the burden on your teams with our fully managed cloud service Save on Costs and Increase your ROI Total Economic Impact of using Confluent • Forrester, March 2022 (Link) STREAM “We are in the business of selling and renting clothes. We are not in the business of managing a data streaming platform… If we had to manage everything ourselves, I would’ve had to hire at least 10 more people to keep the systems up and running.” “Confluent Cloud made it possible for us to meet our tight launch deadline with limited resources. With event streaming as a managed service, we had no costly hires to maintain our clusters and no worries about 24x7 reliability."
  • 35.
    The Confluent Advantage Createa central nervous system for your data across your enterprise Resilient Infinite Elastic Run Consistently Run Anywhere Stream Governance Stream Processing 120+ Connectors Run Unified Apache Kafka® fully managed, and re-architected to harness the power of the cloud Go way beyond Kafka to build real-time apps quickly, reliably, and securely Flexible deployment options across clouds and on-premises that sync in real-time Cloud Native Complete Everywhere Built and supported by the founders of Apache Kafka® Freedom of Choice Committer-driven Expertise Fully Managed Cloud Service Self-managed Software Training Partners Professional Services Enterprise Support End-to-end Security, Privacy, Compliance and Observability
  • 36.
    Why Data Streamingfor Data Products and AI
  • 37.
    Connect and Shareyour Data Everywhere “We are making Confluent the true backbone of BHG, including leveraging over 20 Confluent connectors across both modern, cloud-based technologies and legacy systems, to help integrate our critical apps and data systems together.” 37 Cloud Cloud On-premises Run Anywhere Run Unified Run Consistently Supporting any combination of on-prem, hybrid cloud and multicloud deployments for real-time data interoperability STREAM
  • 38.
    Unlock Real-time ValueFrom Existing Systems and Applications with Ease Pre-built Connectors Use 120+ self-managed and 70+ fully managed zero-code connectors for instant connectivity from and to existing data systems Custom Connectors Bring your own connector and run it confidently and cost-effectively on our fully managed cloud service Native Integrations Enjoy instant access to streaming data directly within popular platforms, so you never have to leave the tool of your choice “Confluent provides exactly what we dreamed of: an ecosystem of tools to source and sink data from data streams. It’s provided us not only with great data pipeline agility and flexibility but also a highly simplified infrastructure that’s allowed us to reduce costs.” Data Diode CONNECT
  • 39.
    Process your dataonce, process your data right Process and Shape Data As It’s Generated For Greater Reuse 39 Lower costs Improve resource utilization and cost-effectiveness by avoiding redundant processing across silos Data reusability Share consistent and reusable data streams widely with downstream applications and systems Data enrichment Curate, filter, and augment data on-the-fly with additional context to improve completeness and accuracy Real-time everywhere Power low-latency applications and pipelines that react to real-time events, as and when they occur PROCESS
  • 40.
    Discover, Understand, and TrustYour Data Streams Where did data come from? Where is it going? Where, when, and how was it transformed? What’s the common taxonomy? What is the current state of the stream? Stream Catalog Increase collaboration and productivity with self-service data discovery Stream Lineage Understand complex data relationships and uncover more insights Stream Quality Deliver trusted, high-quality event streams to the business “Confluent’s Stream Governance suite will play a major role in our expanded use of data in motion and creation of a central nervous system for the enterprise. With the self-service capabilities in stream catalog and stream lineage, we’ll be able to greatly simplify and accelerate the onboarding of new teams working with our most valuable data." GOVERN
  • 41.
    Discover Search and understandthe data of interest through a self-service user interface Access Request and securely grant access to data streams through simple clicks of buttons Build Start working with your data in real-time to pipe it into existing systems or develop real-time apps Explore and Use Real-time Data Products with a Self-service Portal GOVERN
  • 42.
    Without contextualized, trusted,current data… …LLMs can’t drive meaningful value. What is the status of my flight to New York? It is currently delayed by 2 hours and expected to depart at 5 pm GMT. Is there another flight available to the same city that will depart and arrive sooner? What are the seating options and cost? Can your GenAI assistant remember data from an earlier conversation? What is the source of this information? Is this trustworthy? Is it fresh and accurate? How do you augment customer data with real-time data and process them on the fly to provide meaningful insights? The next available flight to New York with United departs later but will arrive faster than your current flight. The only available seats in this flight are first class window seats and costs $1,500.
  • 43.
    Confluent enables real-timeGenAI-powered applications at scale How? • Integrates disparate operational data across the enterprise in real time for reliable, trustworthy use • Organizes unstructured enterprise data embeddings into vector stores • Decouples customer-facing applications from LLM call management to provide reliable, reactive experiences that scale horizontally • Enables LLMs, vector stores, and embedding models to be treated as modular components that can substituted as technology improves
  • 44.
    Enabling Generative AIin the Enterprise Historic Public Data Generative AI Model Intelligent Business-Specific Co-Pilot User Interaction ??
  • 45.
    Enabling Generative AIin the Enterprise Historic Public Data Generative AI Model Intelligent Business-Specific Co-Pilot User Interaction DATA STREAMING PLATFORM
  • 46.
  • 47.
    Data Streaming UseCases for Media & Entertainment INFRASTRUCTURE USE CASES Streaming Data Pipelines Streaming Applications Event-driven Microservices Real-time AI / ML Hybrid / Multicloud Integration Mainframe Integration Middleware / Messaging Modernization Data Warehouse Pipelining CDC Patterns from Systems of Records Log Aggregation …. …. …. …. and more Shift-left Processing & Governance ANTI-CHEAT, CYBERSECURITY, & FRAUD Cheat detection Bot detection Real time automated account bans DDOS detection and mitigation Suspicious login detection Ad fraud detection USER EXPERIENCE & ENGAGEMENT Cross platform experiences User engagement analysis Live engagement Social media integration Next best action/offer Alerts/Notifications Personalized promotions and loyalty programs Event driven marketing Low-latency user experience BUSINESS OPERATIONS Content finance engineering Budget allocation and marketing attribution Marketing campaign analytics Microtransaction analytics AE to fill out slide if we have an active customer engagement. Blue font indicates existing use of Kafka / Confluent at account.
  • 48.
    Challenge: Building acloud-native, 5G, open RAN (ORAN) network that supports a connectivity platform to enable app communications from any cloud to any device. Solution: Adopting Confluent Cloud to support networking and automation across the hybrid cloud and create consumable data products for developers to leverage. Results: DISH Wireless creates data pipelines using streaming data pipelines built with Confluent Cloud. As a result, its connectivity platform can unlock enterprise Industry 4.0 use cases across industries like retail, manufacturing, and automotive. “We're going up orders of magnitude on the volume of data that we want to transact over Confluent. By having a partner like Confluent, we can start generating value from our network at an unimaginable level today.” — Brian Mengwasser, Vice President & Head of MarketPlace & Apps Design Industry: Telecommunications | Geo: Global | Product: Confluent Cloud Click here to learn more.
  • 49.
    Challenge: Decouple agrowing number of backend services, manage large amounts of log data, speed time to innovation for future features. Solution: Implemented Apache Kafka® to connect data across the organization and support the move to microservices. Results: ● Reduced time to deployment ● Achieved massive scalability ● Freed developer resources ● Eliminated workflow interruption “We work with Confluent because it is extremely valuable to have access to experts who know the most about Apache Kafka. It’s that feeling of security that has paid off for us.” — Robert Christ, Principal Engineer Industry: Media & Entertainment | Geo: AMER | Product: Confluent Platform
  • 50.
    Challenge: Improving timeto insights by moving away from conventional batch-based data processing as the sole means of gaining insights and moving toward a stream based approach. Solution: Using streaming data—made possible by Confluent Cloud—to power top-quality, personalized and adaptive experiences for users, inform product roadmap, and drive growth and profitability Results ● Agile decision-making from use of real-time data to monitor and quickly pivot on in-product messaging, campaigns, A/B tests, and experiments ● Adaptive bitrate streaming that ensures seamless playback experience for viewers—without buffering ● Teams can focus on value-added tasks versus managing infrastructure or waiting to access data ● Faster time to market for data products (from days to hours) ● Reduced TCO with Confluent Cloud vs. self-managed Kafka, allowing three FTEs to focus on product development ● Scalability for future use cases, including content delivery optimization and AI/ML “With Confluent, the overarching win is that we’re able to spend more time implementing design and better workflows, making improvements within our current systems, and working on product enhancements. We now have the ability to be driven by business needs.” — Burton Williams, Principal Data Platform Engineer Industry: Video Technology | Geo: AMER | Product: Confluent Cloud Click here to learn more.
  • 52.
  • 53.
    Monoliths to Event-Driven Microserviceswith Confluent Samuele Dell’Angelo Senior Solution Engineer
  • 54.
    54 Tenets for FasterApplication Development Challenges with Monoliths and Interservice Communication How We Do It Confluent Cloud in a nutshell Intro to Stream Processing and Apache Flink Agenda 01 02 03 04 05 06
  • 55.
    Agenda 55 2. Challenges withMonoliths and Interservice Communication 1. Tenets for Faster Application Development 3. How We Do It 4. Confluent Cloud in a nutshell 5. Intro to Stream Processing and Apache Flink
  • 56.
    Tenets for FasterApplication Development 56 Eliminate Dependencies Reduce Technical Debt and TCO Build and Integrate Scalable Apps
  • 57.
  • 58.
    58 Challenges with Monoliths… Monolith ● Slow: Smallest change requires end-end testing ● Unreliable: A simple error can bring down the entire app ● Expensive: Changes to tech stack are expensive. Barrier to adopting new programming frameworks Services Database
  • 59.
    59 Microservices ● Faster releasecycles: Smaller modules that can Independently deployed and upgraded ● Improved Scalability and Reliability: Independently scale services and recover from errors faster ● Developer autonomy: Choose the right programming language and data store for each service Services Databases Services Database … Have Led to the Adoption of Microservices Monolith ● Slow: Smallest change requires end-end testing ● Unreliable: A simple error can bring down the entire app ● Expensive: Changes to tech stack are expensive. Barrier to adopting new programming frameworks
  • 60.
    Two Approaches For Interservice Communication 60 RESTAPIs Synchronous request/response communication Messaging Queues Synchronous or asynchronous communication
  • 61.
    Service 4 Service 3 Challengeswith Using REST APIs for Interservice Communication 61 REST APIs Synchronous request/response Challenges ● Slow Tight coupling means business logic is rewritten to add new services. ● Unreliable Requires costly linear scaling. Unexpected errors are difficult to recover from. ● Expensive High operational overhead of managing services, at scale. Service 2 Service 5 Service 6 Service 1 API API API API API API
  • 62.
    Challenges with UsingMessage Queues for Interservice Communication 62 Messaging Queues Synchronous or asynchronous communication Challenges ● Slow Lack a common structure, leads to inter-team dependencies. ● Unreliable Cannot scale dynamically. Ephemeral persistence. ● Expensive Cannot easily integrate with cloud-native apps. Service 2 Service 5 Service 4 Service 3 Service 6 Service 1 Messaging Queues (No common structure to share data, lacks built-in stream processing, ephemeral message persistence, low fault-tolerance)
  • 63.
    MicroServices -> EventDriven Microservices 63
  • 64.
    Why does itmake sense? Microservices Distributed Semi-loose with API versioning Requires a consistent distributed system Synchronous Event Driven Microservices Central, synchronized through events Completely decoupled (Fire and forget) The streaming platform reduces the dependencies to external system Reactive (asynchronous) State Coupling E2E Testing Execution Model 64
  • 65.
    How We DoIt: 65
  • 66.
    How to Evolveinto Event-Driven Microservices 66 CDC off legacy database / mainframe Leverage Confluent’s connector ecosystem to easily CDC data off legacy mainframes and databases Modernize legacy MQ to event streaming platform Decouple MQ from your application by sending data to Confluent Enrich events with less code using stream processing Join, summarize, and filter event streams in real time, reducing the number of microservices needed Ensure data compatibility with Schema Validation Validate data compatibility and enable evolution of schemas while preserving backwards compatibility 2 3 1 4 Mainframe Microservice 1 Microservice 2 CDC Connector Legacy MQ MQ Connector DB KStreams / KsqlDB / Flink Incompatible Data Schema Registry Validated & Compatible Microservice 1 Microservice 2 Microservice 2 Microservice 1 Microservice 2 Microservice
  • 68.
    Today’s Pattern Write toMy Database Then Use CDC (Write Through) Write to Kafka Then Sink to My Database Commit the Horror That is Dual Writes
  • 69.
    6 9 Suggested approach: Outbox+ CDC Patterns In DDD systems an Aggregate’s internal representation of data may/should be different from the outside contract, In that scenario, outbox pattern can be applied. Application writes data to its table in the database and also to an outbox table within a single Transaction. The outbox table is sourced to Kafka topics. (Write Through)
  • 70.
    7 0 Replace XA transactionsfor multiple database writes ● One Application needs to write to two different database storage engines where XA transactions are not supported. ● Instead of XA transaction, the Application writes data to a single database table, which is sourced to a Kafka topic by a Source connector. Multiple Sink connectors can be used to sink data to different storage engines. Suggested approach: Outbox + CDC Patterns
  • 71.
    Transactional outbox withConfluent 71 Outbox Orders Confluent Platform Connect KsqlDB Ksrteam Relay (no code) Stream processing (code) Applications Stream processing (SQL) Event Broker Connect Other Services
  • 72.
    7 2 Other approach: Databasewrite aside In its default form, this pattern guarantees dual-write for most use cases. However, should the database transaction fail at commit time (say, because the database server has crashed) the write to Kafka cannot be rolled back unless transactions have been enabled. For many use cases, this eventuality will be tolerable as the dual-write can be retried once the failure is fixed, and most event consumers will implement idempotence anyway. However, application programmers need to be aware that there is no firm guarantee. Transactional messaging systems like Kafka can be used to provide stronger guarantees so long as all event consumers have the transactions feature enabled. (Database write aside) Write to a database, then write to Kafka. Perform the write to Kafka as the last step in a database transaction to ensure an atomic dual commit (aborting the transaction if the write to Kafka fails).
  • 73.
    7 3 Temptation: Dual write ●Database transactions and Kafka transactions are separate ● To perform them atomically would need to be done as a distributed transaction ● You can use Spring Transaction Manager ○ Chaining Kafka and Database Transactions with Spring Boot: An In-Depth Look ○ Examples of Kafka Transactions with Other Transaction Managers (Dual write) Write to a database and to Kafka using an atomic approach. KAFKA DATABASES Write APP Write
  • 74.
    74 Are you safewith this approach?
  • 75.
    75 Kafka at thismoment doesn’t support 2 Phase Commit
  • 76.
  • 77.
    KIP-939: Support Participation in Two Phase Commit Enablesconsistent and atomic update of Kafka and other transaction stores 3. On the Horizon PRODUCER APP PRODUCER APP COMMIT COMMIT PREPARED ABORT ABORT
  • 78.
    Solving 2PC withKIP-939 Red Flags InitProducerId must be called on client restarts, it currently aborts any ongoing transactions Transaction.max.timeout.ms can abort transactions independent of the transaction coordinator
  • 79.
    Example Flow 7 9 KAFKA DATABASES 2 5 2 3 1 APP 4 1. Commitor Abort Previous Transaction in Kafka (if any) 2. Start New Transactions in Kafka and the Database; Write App Data 3. 2PC Prepare Phase: Prepare Kafka Transactions 4. Persist 2PC Decision 5. 2PC Commit Phase: Commit Kafka Transactions
  • 80.
    FLIP-319 – Integratingwith Kafka 2PC Transactions Hardens EOS Across Flink and Kafka Simplifies Upgrading Kafka Clients Used by Flink
  • 81.
  • 82.
  • 83.
  • 84.
    E-Commerce Application (Monolith) 84 Catalog Order Payment On-PremMonolith (E-commerce App) Load Balancer Client/Web Application Writes Reads Writes Reads
  • 85.
    Step 1: RouteReads to Catalog Microservice 85 Catalog Order Payment On-Prem Monolith (E-commerce App) Load Balancer Client/Web Application Writes to Catalog module Other Writes and Reads Writes Reads Writes CDC Source Connector Streams (Reads) Reads from Microservice Partially replaces Catalog Module Cloud Native Catalog Microservice
  • 86.
    Step 2: FullyReplace Catalog Module 86 Order Payment On-Prem Monolith (E-commerce App) Load Balancer Client/Web Application Other Writes and Reads Writes Reads Writes CDC Source +Sink Connector Cloud Native Catalog Microservice Streams (Reads) Reads from Microservice Reads Streams (Writes) Writes to Microservice Third Party Consumer Apps
  • 87.
    Challenge: eBay Koreawanted to enable continued growth across multiple e-commerce platforms that had been in place for more than 15 years. Solution: eBay Korea leverages Confluent to help migrate from a monolithic architecture to a microservices architecture that enables more independence, greater flexibility, and improved scalability. Results: ● Simplified maintenance and operations for improved productivity ● Stable, high-throughput streaming ● Rapid support ● Faster time to market “Although we had some experience running open source Kafka ourselves, we recognized that we needed a trusted partner to help us resolve issues that we’d encounter with a larger-scale deployment. Confluent not only had a local office to provide that support, they also had the best known, verified solution, which was what we wanted for our mission-critical e-commerce platforms.” — Hudson Lee, Leader of Platform Engineering
  • 88.
  • 89.
    The Confluent ProductAdvantage 89 Re-imagined Kafka experience for the Cloud Be everywhere our customers want to be COMPLETE EVERYWHERE CLOUD NATIVE Enable developers to reliably & securely build next-gen apps faster
  • 90.
    90 Cloud-native data streamingplatform built by the founders of Apache Kafka® KORA: THE KAFKA ENGINE FOR THE CLOUD STREAM Everywhere Be everywhere our customers want to be Cloud-native Re-imagined Kafka experience for the Cloud Complete Go above and beyond Kafka with a complete data streaming platform CONNECT GOVERN PROCESS Confluent Data Streaming Platform
  • 91.
    Kora Architecture We spent5 million+ hours to re-invent Kafka into a truly cloud-native engine NETWORK COMPUTE OBJECT STORAGE CUSTOMERS AZ AZ AZ Cells Cells Cells Multi-Cloud Networking & Routing Tier Metadata Durability Audits Data Balancing Health Checks Real-time feedback data GLOBAL CONTROL PLANE S E C U R I T Y
  • 92.
    THE KAFKA ENGINEFOR THE CLOUD STREAM RESILIENT COST EFFECTIVE 10X FAST ELASTIC 16X 30X
  • 93.
    Using fully managedconnectors is the fastest, most efficient way to break data silos Self-managed connector Accelerated time-to-value • Increased developer productivity • Reduced operational burden ● Pre-built but requires manual installation / config efforts to set-up and deploy connectors ● Perpetual management and maintenance of connectors that leads to ongoing tech debt ● Risk of downtime and business disruption due to connector / Connect cluster related issues Fully managed connector Custom-built connector ● Streamlined configurations and on-demand provisioning of your connectors ● Eliminates operational overhead and management complexity with seamless scaling and load balancing ● Reduced risk of downtime with Confluent Cloud’s 99.95% SLA for all your mission critical use cases ● Costly to allocate resources to design, build, test, and maintain non-differentiated data integration components ● Delays time-to-value, taking up to 3-6+ engineering months to develop ● Perpetual management and maintenance increases tech debt and risk of downtime
  • 94.
    Connect your entirebusiness with just a few clicks 70+ fully managed connectors Amazon S3 Amazon Redshift Amazon DynamoDB Google Cloud Spanner AWS Lambda Amazon SQS Amazon Kinesis Azure Service Bus Azure Event Hubs Azure Synapse Analytics Azure Blob Storage Azure Functions Azure Data Lake Google BigTable
  • 95.
    Stream Governance isprovided fully managed on Confluent Cloud Stream Quality Deliver trusted, high-quality data streams to the business Stream Catalog Increase collaboration and productivity with self-service data discovery Stream Lineage Understand complex data relationships and uncover more insights (Schema Registry & Validation) Discover, understand, and trust your data streams with the industry’s only fully managed governance suite for Apache Kafka® Stream Governance
  • 96.
    schema registry SCHEMA METADATASCHEMA METADATA Producer App 3 ... Producer App 1 Producer App 2 Consumer App 3 ... Consumer App 1 Consumer App 2 UNIVERSAL LANGUAGE & QUALITY CONTROL Stream Quality Deliver trusted, high-quality data streams to the business Avro, Protocol Buffers, and JSON serialization formats supported validate schemas before producing validate schemas before consuming Schema Registry is available with a 99.95% Uptime SLA Maintain trusted, compatible data streams across cloud and hybrid environments with shared schemas that sync in real time, compatibility rules, and schema references plugin
  • 97.
    Intro to StreamProcessing and Apache Flink
  • 98.
    Real-time processing Power low-latencyapplications and pipelines that react to real-time events and provide timely insights Data reusability Share consistent and reusable data streams widely with downstream applications and systems Data enrichment Curate, filter, and augment data on-the-fly with additional context to improve completeness, accuracy, & compliance Efficiency Improve resource utilization and cost-effectiveness by avoiding redundant processing across silos “With Confluent’s fully managed Flink offering, we can access, aggregate, and enrich data from IoT sensors, smart cameras, and Wi-Fi analytics, to swiftly take action on potential threats in real time, such as intrusion detection. This enables us to process sensor data as soon as the events occur, allowing for faster detection and response to security incidents without any added operational burden.” Effortlessly filter, join, and enrich your data streams with Apache Flink
  • 99.
    Blob storage 3rd partyapp Databases Data Warehouse Database SaaS app Low latency apps and data pipelines Consistent, reusable data products Optimized resource utilization Process data streams in-flight to maximize actionability, fidelity, and portability
  • 100.
    Pre-Packaged stream processing workloadssuch as: ● deduplication topic ● masking fields EVENT-DRIVEN APPLICATIONS Flink Actions
  • 101.
    C price>lag(price) D price<lag(price) C price>lag(price) B price<lag(price) A Double Bottom Period &Volume Price Develop applications using fine-grained control over how time progresses and data is grouped together using: ● Hopping, tumbling, session windows ● OVER aggregations ● Pattern matching with MATCH_RECOGNIZE EVENT-DRIVEN APPLICATIONS Recognize patterns and react to events in a timely manner
  • 102.
    Get up-to-date resultsto power dashboards or applications requiring continuous updates using: ● Materialized views ● Temporal analytic functions ● Interactive queries Account Balance A $15 B $2 C $15 Account A, +$10 Account B, +$12 Account C, +$5 Account B, -$10 Account C, +$10 Account A, -$5 Account A, +$10 Time REAL-TIME ANALYTICS Analyze real-time data streams to generate important business insights
  • 103.
    Create new enrichedand curated streams of higher value using: ● Data transformations ● Streaming joins, temporal joins, lookup joins, and versioned joins ● Fan out queries, multi-cluster queries 103 t1, 21.5 USD t3, 55 EUR t5, 35.3 EUR t0, EUR:USD=1.00 t2, EUR:USD=1.05 t4: EUR:USD=1.10 t1, 21.5 USD t3, 57.75 USD t5, 38.83 USD Currency rate Orders STREAMING DATA PIPELINES Build streaming data pipelines to inform real-time decision making
  • 104.
    INSERT INTO enriched_reviews SELECTid , review , invoke_openai(prompt,review) as score FROM product_reviews ; K N B Kate 4 hours ago This was the worst decision ever. Nikola 1 day ago Not bad. Could have been cheaper. Brian 3 days ago Amazing! Game Changer! K N B Kate ★★★★★ 4 hours ago This was the worst decision ever. Nikola ★★★★★ 1 day ago Not bad. Could have been cheaper. Brian ★★★★★ 3 days ago Amazing! Game Changer! The Prompt “Score the following text on a scale of 1 and 5 where 1 is negative and 5 is positive returning only the number” DATA STREAMING PLATFORM Enrich real-time data streams with Generative AI directly from Flink SQL Next
  • 106.