The Top 5 Apache Kafka Use Cases and Architectures in 2022

The Top 5 Use Cases and Architectures for Data in Motion in
2022
Kappa Architecture, Omnichannel, Multi-Cloud, Edge Analytics, and Real-time Cybersecurity
Kai Waehner
Field CTO
kai.waehner@confluent.io
linkedin.com/in/kaiwaehner
@KaiWaehner
confluent.io
kai-waehner.de

@KaiWaehner www.kai-waehner.de
https://www.gartner.com/en/information-
technology/insights/top-technology-trends

Cloud
Machine
Learning
Mobile
Data in
Motion
Rethink
Decision Making
Rethink
User Experience
Rethink
Data
Rethink
Data Centers

Real-time Data in Motion beats Slow Data.
Transportation
Real-time sensor
diagnostics
Driver-rider match
ETA updates
Banking
Fraud detection
Trading, risk systems
Mobile applications /
customer experience
Retail
Real-time inventory
Real-time POS reporting
Personalization
Entertainment
Real-time
recommendations
Personalized
news feed
In-app purchases

This is a fundamental paradigm shift...
5
Infrastructure
as code
Data in motion
as continuous
streams of events
Future of the
datacenter
Future of data
Cloud
Event
Streaming

Apache Kafka is the Platform for Data in
Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time Alerting
System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
6

The Top 5 Use Cases and Architectures for Data in Motion in
2022
1) The Kappa Architecture
2) Hyper-personalized Omnichannel
3) Multi-Cloud Deployments
4) Edge Analytics
5) Real-time Cybersecurity

Lambda Architecture
Option 1: Unified serving layer
9
Data
Source
Real-Time Layer
(Data Processing in Motion)
Batch Layer
(Data Processing at Rest)
Serving Layer
Real-Time App
Batch App
ms
min/hr

10
Data
Source
Real-Time Layer
Batch Layer
Real-time Query
Mixed Query
ms
min/hr
Speed
View
Batch
View
Batch Query
Lambda Architecture
Option 2: Separate serving layers

Concerns with the Lambda Architecture
11

12
Data
Source
Real-Time Layer
Real-Time App
Storage
Batch App
Storage
ms
min/hr
Storage
Kappa Architecture
One pipeline for real-time and batch consumers

Tiered Storage for Kafka
13

Kappa @ Uber
14

Kappa @ Shopify
15
Kappa Building Blocks
The Log (Kafka)
Durability with Topic Compaction and Tiered Storage
Consistency via Exactly-Once Semantics (EOS)
Data Integration via Kafka Connect
Elasticity via dynamic Kafka clusters
Streaming Framework (Kafka Streams / Flink)
Reliability and scalability
Fault tolerance
State management
Sinks
Update/Upsert for simplified design:
RDBMS, NoSQL, Compacted Kafka Topics
Append-only: Regular Kafka Topics, Time Series

Kappa @ Disney
16
www.kai-waehner.de | @KaiWaehner | Streaming Machine Learning without a Data Lake

Benefits of the Kappa Architecture
The Kappa architecture leverages a single source of truth with a focus on simplicity in the
enterprise architecture
• Improve streaming to handle all the cases
• One codebase that is always in synch
• One set of infrastructure and technology
• The heart of the infrastructure is real-time, scalable, and reliable
• Improved data quality with guaranteed ordering and no mismatches
• No need to re-architect for new use cases, just connect new consumers (real-time, near
real-time, batch, RPC)
• Kappa is NOT a free lunch – know the trade-offs and best practices
17

Kappa is NOT a free lunch
18

Kappa Concerns Solved
• Data availability / retention
 Compacted Topics, Tiered Storage
• Data consistency and fault-tolerance
 Exactly-once semantics, Multi-Region Clusters, Cluster Linking
• Handling late-arriving data
 State management in the streaming application, proper data sinks, replay with
guaranteed ordering and timestamps
• Data reprocessing and backfill
 Dynamic clusters, stateful applications (Kafka Streams, ksqlDB, external stream
processing framework like Apache Flink)
• Data integration
 Kafka Connect for sources and sinks, clients for any language, REST Proxy (real-time
but also batch and RPC
19

The New Business Reality
Technology is the business
Innovation required for survival
Yesterday’s data = failure
Modern, real-time data
infrastructure is required.
Technology was
a support function
Innovation required for growth
“Good enough” to run on
yesterday’s data

Real-time automation of customer interactions
Improved Shipping and Delivery
Methods
Customer-Driven
In-Store Experiences
Hybrid model
Shopping
Social Influencers / Virtual Reality Shopping:
Journey-focused innovation
General Trends:
● Highly competitive market, work to thin margins
● Moving from High Street (brick & mortar) to Online (Omni-Channel)
● Personalized Customer Experience - optimal buyer journey
Customer
Experience
(CX) Operational
Efficiencies
New
Business
Models
Disruptive
Trends
in
Retail
Warehouse logistics teams aligned with
real-time, in-store demands
Automating the supply chain and core
business processes
Data-Driven
Business Decisions and
Personalized Promotions

“Walmart is a $500 billion in revenue
company, so every second is worth millions of
dollars. Having Confluent
as our partner has been invaluable.
Kafka and Confluent are the
backbone of our digital
omnichannel transformation
and success at Walmart.”
VP of Walmart Cloud

Real-Time Inventory System
https://www.confluent.io/blog/walmart-real-time-inventory-management-using-kafka/
https://www.confluent.io/kafka-summit-san-francisco-2019/when-kafka-meets-the-scaling-and-reliability-needs-of-worlds-largest-retailer-a-walmart-story/
● Investment in Kafka and Confluent has helped topline company
growth
● 8,500 nodes processing 11 billion events per day
● Deliver an omnichannel experience so every customer can
shop the way they want to

Context-specific Customer 360
25
Electrical retailer
Hyper-personalized online retail
experience, turning each
customer visit into a one-on-one
marketing opportunity
Correlation of historical customer
data with real-time digital signals
Maximize customer satisfaction
and revenue growth, increased
customer conversions
https://www.confluent.io/customers/ao/

Dick’s Sporting Goods
26
America’s largest sporting goods retail company
Focused on helping athletes achieve their personal best
Reshape the way athletes gain access to context-specific product
information in real time for a more seamless purchasing
experience online and in stores
Handle pricing and promotions, marketing, and athlete services in
real time to ensure a consistent omnichannel experience and
positive athlete service interaction
Fully-managed multi-cloud strategy with Confluent Cloud for
improved time-to-market and reduced operations cost.
confluent.io/customers/dicks-sporting-goods

Omnichannel Retail
Time
P
C3 C2
C1
Sales Talk on site in
Car Dealership
Right now
Location-based
Customer Action
Customer 360
(Website, Mobile App, On Site in Store, In-Car)
Car Configurator
10 and 8 days ago
Context-specific
Marketing Campaign
90 and 60 days ago

Live commerce with real-time data correlation
including integration of CRM, loyalty, inventory, chatbots, location-based services, etc.
28

Omnichannel Retail
Time
P
C3 C2
C1
Machine Learning
Context-specific
Recommendations
Location-based
Customer Action
Customer 360
(Business Intelligence, Machine Learning)
Machine Learning
Train Recommendation Engine
Reporting
All Customer Interactions

Spaghetti: Data architectures are often complex
31

Kafka provides a solution:
An immutable stream of facts with the freedom to act, adapt, and change
32
Kafka

Domain
33
Data
Product
Data Mesh: A new technology-agnostic decentralized implementation
pattern
Data Mesh
Data ownership
by domain
Data as
a product
Data available anywhere,
self serve
Data governed
wherever it is

34
Mesh is one logical cluster. Data product has another.
Data
Product
Data Product has its own
cluster for internal use

35
With stream processing the real-time applications are decentralized
Data
Product
STREAM
PROCESSOR
ksqlDB
Query is the interface to
the mesh
Events are the interface
to the mesh

36
Operational and Data Product Streaming Planes with
Cluster Linking

Data Mesh Example: Hybrid Multi-Cloud Architecture
37
Data Engineers
Data Scientists
Data Architects Operators
Architects
SMEs
Data Governance
Shared Services
Application team
Generalist Eng
Generalists Eng
Specialized / Legacy Engineers

Kafka as a Service – Fully Managed?
Infrastructure
management
(commodity)
Scaling
● Upgrades (latest stable version of Kafka)
● Patching
● Maintenance
● Sizing (retention, latency, throughput, storage, etc.)
● Data balancing for optimal performance
● Performance tuning for real-time and latency requirements
● Fixing Kafka bugs
● Uptime monitoring and proactive remediation of issues
● Recovery support from data corruption
● Scaling the cluster as needed
● Data balancing the cluster as nodes are added
● Support for any Kafka issue with less than X minutes response time
Infra-as-a-Service
Harness full power of Kafka
Kafka-specific
management
Platform-as-a-Service
Evolve as you need
Future-proof
Mission-critical reliability
Most Kafka-as-a-Service offerings are partially-managed
Kafka as a Service should be a serverless experience with consumption-based pricing!

Data Governance: Tracking data lineage with Streams in real-time
39
• Lineage must work across domains and data products—and systems, clouds, data centers.
• Event streaming is a foundational technology for this.
On-premise

What is the “Edge” for Kafka?
• Edge is NOT a data center
• Kafka clients AND the Kafka broker(s)
• Offline business continuity
• Often 100+ locations
• Low-footprint and low-touch
• Hybrid integration

Edge Use Cases with Low Latency Requirements
https://www.youtube.com/watch?v=A9DDe0alvGo

Low Latency 5G Use Cases for Edge and Hybrid
Cloud
with AWS Wavelength (based on AWS Outposts) and Confluent

CRM
3rd party
payment
provider
Context-specific
real-time upsell
Customer data
Payment processing and
fraud detection as a service
Manager
Get report
API
Customer Customer
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Hybrid Retail Architecture

Point of Sale
(POS) Loyalty
System
Local Inventory
Management
Payment Discount
Customer
data
Train
schedule
Payment
data
Loyalty
information
Streams of real time events
Global Inventory
Management
Event Streaming at the Edge
in the Smart Retail Store
Item Availability

Disconnected Edge
Time
P
C3 C2
C1
Context-specific
Advertisement
Real-time
(Milliseconds)
Location-based
Customer Action
Always on (even “offline”)
Replayability
Reduced traffic cost
Better latency
Payment Processing
Near Real-time
(Seconds)
Replication to Cloud
Batch
(Depending on Network Bandwidth)

Ship-Shore Highway – Swimming Retail Stores
https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/

Devon Energy Corporation
Oil & Gas Industry
Improve drilling and well completion operations
Edge stream processing/analytics + closed-loop control ready
Replication to the cloud in real-time at scale
Vendor agnostic (pumping, wireline, coil, offset wells, drilling
operations, producing wells
Cloud agnostic (AWS, GCP, Azure)

What is Cybersecurity?
Protection of computer systems and networks from information disclosure and theft
Web Scraping, hackers, criminals, terrorists, state-sponsored and state-initiated actors
50

Supply Chain Attack
Targeting less-secure elements in the supply chain
51
https://www.nortonrosefulbright.com/en/knowledge/publications/dfa3603c/six-degrees-of-separation-cyber-risk-across-global-supply-chains
https://www.reuters.com/article/us-tmobile-dataprotection-idUSKCN0RV5PL20151002

Real-time Data in Motion beats Slow Data.
Security
Access control and encryption
Regulatory compliance
Rules engine
Security monitoring
Surveillance
Cybersecurity
Risk classification
Threat detection
Intrusion detection
Incident response
Fraud detection

Data in Motion
The Backbone for Cybersecurity
Industria
l OT
Enterpris
e IT
Consumer
IoT
Logs Personal
Sensors Security
Streams of real time
events
53
Connected
Vehicles
Cyber
Security
Continuous
Data Correlation
Monitoring
Alerting
Proactive Actions

End-to-End Cybersecurity
with the Kafka Ecosystem
Personel
Crew, Cargo
Vessel
Fuel Consumption, Speed,
Planned Maintenance
Tracking
Position, Course, Weather, Draft
Drone or Satellite Relay
COMMs Resilient Kafka
Edge Analytics
Data
Integration
Streaming Analytics
Machine Doing
On-Prem Systems
Bi-Directional Hybrid Cloud Replication
ON SHORE
ON PREM
Staging, Filtering
Shore Edge Analytics

SIEM / SOAR
Situational Awareness
Operational Awareness
Intrusion Detection
Signals and Noise
Signature Detection
Incident Response
Threat Hunting & Intelligence
Vulnerability Management
Digital Forensics
…
was not built for cybersecurity!

Integrate with all legacy and modern interfaces
Record, filter, curate a broad set of traffic streams
Let analytic sinks consume just the right amount of data
Drastically reduce the complexity of the enterprise architectures
Drastically reduce the cost of SIEM / SOAR deployments
Add new analytics engines
Add stream-speed detection and response at scale in real-time
Add mission-critical (non-) security-related applications
…
is the backbone for cybersecurity!

Confluent Sigma
Sigma Stream Processors
Zeek Data and
Detections Viewer
Sigma Rule Editor
sigma rules topic
DNS
dns
detections
topic
dns topic
rule parsing,
filtering,
aggregation,
windowing
sigma
rules
cache
CONN
DHCP
HTTP
SSL
x509
Zeek Data

Cyber Intelligence Platform
leveraging Kafka Connect, Kafka Streams, Multi-Region Clusters (MRC), and more…
https://www.intel.com/content/www/us/en/it-management/intel-it-best-practices/modern-scalable-cyber-intelligence-platform-kafka.html

How does
Confluent
help?

The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka
60

I N V E S T M E N T & T I M E
V
A
L
U
E
3
4
5
1
2
Event Streaming Maturity Model
Initial Awareness
/
Pilot (1 Kafka
Cluster)
Start to Build
Pipeline / Deliver
1 New Outcome
(1 Kafka Cluster)
Mission-Critical
Deployment
(Stretched,
Hybrid, Multi-
Region)
Build Contextual
Event-Driven Apps
(Stretched,
Hybrid, Multi-
Region)
Central Nervous
System
(Global Kafka)
Product, Support, Training, Partners, Technical Account Management...
61

Car Engine Car Self-driving Car
Confluent completes Apache Kafka. Cloud-native.
Everywhere.

Confluent... Complete. Cloud-native. Everywhere.
Freedom of Choice
Committer-driven Expertise
Open Source | Community licensed
Fully Managed Cloud Service
Self-managed Software
Training Partners
Enterprise
Support
Professional
Services
ARCHITECT
OPERATOR
DEVELOPER EXECUTIVE
Apache Kafka
Dynamic Performance & Elasticity
Self-Balancing Clusters | Tiered Storage
Flexible DevOps Automation
Operator | Ansible
GUI-driven Mgmt & Monitoring
Control Center | Proactive Support
Event Streaming Database
ksqlDB
Rich Pre-built Ecosystem
Connectors | Hub | Schema Registry
Multi-language Development
Non-Java Clients | REST Proxy
Admin REST APIs
Global Resilience
Multi-Region Clusters | Replicator
Cluster Linking
Data Compatibility
Schema Registry | Schema Validation
Enterprise-grade Security
RBAC | Secrets | Audit Logs
TCO / ROI
Revenue / Cost / Risk Impact
Complete Engagement Model
Efficient Operations
at Scale
Unrestricted
Developer Productivity
Production-stage
Prerequisites
Partnership for Business
Success

Kai Waehner
Field CTO
kai.waehner@confluent.io
@KaiWaehner
kai-waehner.de
confluent.io
linkedin.com/in/kaiwaehner
Questions?
Feedback?
Let’s connect!

The Top 5 Apache Kafka Use Cases and Architectures in 2022

More Related Content

What's hot

Similar to The Top 5 Apache Kafka Use Cases and Architectures in 2022

More from Kai Wähner

Recently uploaded

The Top 5 Apache Kafka Use Cases and Architectures in 2022

Editor's Notes