Digital Business Transformation in the Streaming Era

Digital Business
Transformation in the
Streaming Era

2© 2018 Attunity 2© 2017 Attunity
TODAY’S SPEAKERS
Mike Boyarski
Sr Director Product
Marketing
MemSQL
Clive Bearman
Director Product
Marketing
Attunity
James Kobielus
Lead Analyst for Data Science,
Deep Learning, and App Dev
Wikibon

AGENDA
James – Digital Business Transformation Perspective
Clive – A Path to Transformation
Mike – Working Example
Q&A

Digital BusinessTransformation in
the Streaming Era
June 28, 2018

Digital transformation is all about customer
intimacy.
“The purpose of
business is to create
and keep a customer.”
“Digital businesses use data
continuously, statefully,
contextually, and intelligently to
create and keep a customer.”
©Wikibon 2018
Peter Drucker

All-streaming architectures enable continual
customer engagement.
• LIVE: Digital businesses catch and keep
customers via “live environments”:
mobile, edge, embedded, Internet of
Things, and serverless computing.
• INTERACTIVE: All of those live
environments depend on stream
computing: low-latency, in-memory,
continuous, on-demand, and interactive.
• UBIQUITOUS: Stream computing drives
speedier business results through
delivery of live intelligence into all
customer-facing engagements and
enabling back-end processes.
Image: Flicker CC
©Wikibon 2018

Customer engagement rides on layered streaming.
• CONTINUOUS: Low-latency event
ingestion.
• STATEFUL: Continuous PLUS
persistence and state
management.
• CONTEXTUAL: Stateful PLUS
context, graph, and transaction
management.
• INTELLIGENT:Contextual PLUS in-
stream machine learning, deep
learning, natural language
processing, artificial intelligence,
and advanced analytics
Image: VideoCoin
©Wikibon 2018

Digital transformation is powering demand for
stream processing solutions.
Source: Wikibon’s 2018 Big Data Analytics
Trends and Forecast
 Streaming platforms will become the
core focus of most customer
engagement solutions by 2027.
 Ubiquitous adoption of mobile,
embedded, IoT, edge, microservices,
and serverless computing will sustain
growth in the stream computing
market.
 Enterprises will continue to grow their
investments in streaming solutions
while converging those investments
with their big data at-rest
environments, including Hadoop,
NoSQL, and RDBMSs.
©Wikibon 2018

Streaming will revolutionize the entire big data
ecosystem.
• Over the coming decade, the data store as we used to
know it will be ancient history, in a world where most
customer engagement rides on streaming, in-memory,
and serverless infrastructures.
• Batch-oriented big data deployments are giving way to
more completely real-time, streaming, low-latency end-
to-end environments.
• Kafka, Flink, and Spark Streaming are becoming the
principal stream computing platforms.
• Public cloud providers have put streaming, edge, IoT,
serverless, and real-time offerings at the heart of their big
data solutions.
• Real-time continuous DevOps workflows have become the
heart of the app-dev lifecycle.
Image: Pixabay
©Wikibon 2018

Streams, not stores, are the transformational platforms
driving 21st century business.
• Modernize: move your digital
business infrastructure to a low-
latency, in-memory, stream-
computing fabric.
• Standardize: deploy your
streaming environment on open-
source platforms, especially
Apache Kafka,Apache Flink, and
Spark Streaming.
• Operationalize: build a robust,
scalable, stateful, continuous,
and intelligent streaming
backbone. ©Wikibon 2018
Image: Pixabay

MODERN DATA INTEGRATION
IN THE STREAMING ERA

JIM’S TAKEAWAYS
MODERNIZE STANDARDIZE OPERATIONALIZE

13© 2018 Attunity
MAINE SAYING
“You Can’t Get
There From Here.”

CLOUDSTREAMINGDATA LAKES
ON PREMDATA WAREHOUSE BATCH
WHY YOU CANT GET THERE FROM HERE

CLOUDSTREAMINGDATA LAKES
ON PREM
DATA
WAREHOUSE
BATCH
MODERN DATA PLATFORMS

GETTING TO THERE FROM HERE
HADOOP
RDBMS
DATA
WAREHOUSE
FILES
MAINFRAME
STREAMING

CDC IS THE KEY TO STREAMING
HADOOP
RDBMS
DATA
WAREHOUSE
FILES
MAINFRAME
TRANSFER STREAMING
INCREMENTALCDC
Application
Users
Sensor
Applications

CDC WITH ATTUNITY REPLICATE
INCREMENTAL
HADOOP
RDBMS
DATA
WAREHOUSE
FILES
MAINFRAME
HADOOP
RDBMS
DATA
WAREHOUSE
STREAMING
FILES
CDC
TARGET SCHEMA
CREATION
HETEROGENEOUS
DATA TYPE MAPPING
BATCH TO CDC
TRANSITION
DDL CHANGE
PROPAGATION
FILTERING
TRANSFORMATIONS

ATTUNITY REPLICATE
Universal
Connectivity
No Coding.
Simple GUI
Automated
Replication
Agentless
Configuration
Fast Data
Transfer

Real-time, no-impact CDC Manage many end points
and streams at scale
Stream optimized – schema
evolution, multi-topic, multi-
partition
ATTUNITY DIFFERENTIATORS – STREAMING

Automate to reduce
$
Leverage reusable
data pipelines
Build repeatable
process
ROADMAP IS CLEAR
Think hybrid (cloud &
on-prem)
Choose best-of-
breed components
Embrace streaming
Add CDC to your
stack
Align skills & close
gap
MODERNIZE STANDARDIZE OPERATIONALIZE

#1 provider of change
data capture (CDC)
Support most sources
with best performance
and least impact
#1 cloud database
migration technology
Already moved over
55,000 databases
to Cloud platforms
like AWS
#1 in ease of use
Designed to accelerate
deployments by data
architects and
DBAs instead of
developers
The leading platform for delivering data efficiently and in
real-time to data lake, streaming and cloud architectures
ATTUNITY

25© 2018 Attunity
FINANCIAL
SERVICES
MANUFACTURING/
INDUSTRIAL
HEALTH
CARE
GOVERNMENT TECHNOLOGY /
TELECOM
RETAIL OTHER
INDUSTRIES
2000 CUSTOMERS AND HALF THE FORTUNE 100

Trusted by Microsoft
with 3 OEMs,
bundled inside
SQL Server
Trusted by Amazon
(AWS) with strategic
partnership for cloud
database migration
Trusted by IBM and
Oracle with respective
OEMs of Attunity
technology
Trusted by Teradata
and HP as resellers for
data warehouse and
analytics
Trusted by
global system
integrators
Trusted by over
2000 customers for
commitment, flexibility
and speed
2000+
Trusted by SAP as
certified solution in use
with over 200 SAP
customers
Trusted by big data
leaders for data lake
solutions
Trusted by IBM and
Oracle with respective
OEMs of Attunity
technology
Trusted by Teradata
and HP as resellers for
data warehouse and
analytics
PARTNER OF CHOICE

Automated, universal and
real-time data delivery
Accelerate creation of analytics-
ready data structures
ATTUNITY REPLICATE ATTUNITY COMPOSE
ATTUNITY ENTERPRISE MANAGER
Intelligent management, metadata & control
THE ATTUNITY PLATFORM

Database Modernization
for Digital Business

29
MemSQL at a Glance
Mission Enable any company to be a real-time enterprise
Product Top Ranked Database and Data Warehouse
Customers

30
Database Challenges Impacting Digital Business Initiatives
Lengthy Query Execution
Limited User Access
Slow Data Loading

31
The Digital Enterprise Requires Performance
Fast Queries
Scalable SQL
Real-time dashboards
1 Trillion row scan a second
Scalable User Access
Multi-threaded processing
Converged transactions and analytics
Scale-out for performance
Live Loading
Stream data
ACID transactions
Multiple sources

32
MemSQL Architecture
Operational Database and Data Warehouse in One
Historical Data
Disk-optimized tables
with compression for
fast analytic queries
Live Data
Memory optimized tables
for analyzing real-time
events
Streaming Ingest
Real-time data pipelines
with exactly-once
semantics

33
Transforming profitability analysis of customer logistics from
weekly to daily using modern data streaming architectures
+

OLTP Sources DashboardsData Integration Data Mart
SAP on Oracle
SQL Server
SAP Data Services MySQL
Tableau
Case Study: Slow Performing Analytics with Batch ETL

SAP on Oracle
SQL Server
Challenges:
• 24 hour ETL process
• Minimize operational
database impact
Tableau

SAP on Oracle
SQL Server
Challenges:
• 24 hour ETL process
• Single threaded
operation
Tableau
Challenge:
Slow running queries

Case Study: Faster Analytics with Live Loading
OLTP Sources Dashboards
SAP on Oracle
SQL Server
Tableau
Why MemSQL?
- Fast distributed data ingestion
- 22 hrs ingestion to 43 minutes
SAP Data Services

Case Study: Faster Analytics with Live Loading
OLTP Sources Dashboards
SAP on Oracle
SQL Server
Tableau
SAP Data Services
Why MemSQL?
- Fast queries with no pre-
aggregates
- 20x dashboard acceleration

+
RESULTS
39
30x faster
Data latency
dropped from
22 hours 
43 minutes
80x query response
improvement over
mySQL
Profitability analysis
of customer
logistics was
transformed from
weekly to daily

40
Accelerating the frequency and accuracy of fraud detection on live
operational systems with streaming architectures
+
US Oil and Gas
Exploration Company

41
Case Study: Invoice Analysis Performance on Oracle
Oracle
Step 1
Vendor Invoice
committed to
Oracle
Financials
Step 2
Analyst checks
for duplication
and invoice
accuracy
Step 3
Batch rule job
runs at night
comparing all
invoices for
duplication
Step 4
Invoice cleared for
payment. The compare
job fails often enough
and invoice has to be
paid out. For Duplicates,
clawback needs to
happen.
ERP Source Invoice Audit Reports

42
Oracle
Step 1
Vendor Invoice
committed to
Oracle
Financials
Step 2
Analyst checks
for duplication
and invoice
accuracy
Step 3
Batch rule job
runs at night
comparing all
invoices for
duplication
Step 4
Invoice cleared for
clawback needs to
happen.
Challenges:
- Batch job took 12 hrs to run on
Oracle
- Oracle tuning unable to perform
- Unable to migrate from Oracle
due to core operational system
ERP Source Invoice Audit Reports

43
ERP Source
Oracle
Step 1
Vendor Invoice
committed to
Oracle
Financials
Step 2
Analyst checks
for duplication
and invoice
accuracy
Step 3
Batch rule job
runs at night
comparing all
invoices for
duplication
Step 4
Invoice cleared for
clawback needs to
happen.
Challenges:
- Batch job took 12 hrs to run on
Oracle
- Oracle tuning unable to perform
- Unable to migrate from Oracle
due to core operational system
Invoice Audit Reports
Challenge:
- Incorrect invoice payments cost
millions per year

44
Case Study: Continuous Invoice Analysis for Savings
ERP Source
Oracle
Step 3
Analyst checks MemSQL for
duplicate invoices in real-time
Step 2
CDC job replicates
Oracle to MemSQL
Step 4
Faster invoice
clearance with
reduced clawbacks
and duplications
Fraud Reports
Step 1
Vendor Invoice
created in the
system.
Why MemSQL?
• Reduced 12 hr batch process to 22 seconds
• Invoice validated at time of submission

45
Case Study: Continuous Invoice Analysis for Savings
ERP Source
Oracle
Step 3
Analyst checks MemSQL for
duplicate invoices in real-time
Step 2
CDC job replicates
Oracle to MemSQL
Step 4
Faster invoice
clearance with
reduced clawbacks
and duplications
Fraud Reports
Step 1
Vendor Invoice
created in the
system.
Why MemSQL?
• Reduced 12 hr batch process to 22 seconds
• Invoice validated at time of submission
Why MemSQL?
• Dramatic reduction in
clawbacks
• ROI realized in months

MemSQL and Attunity – Enabling Digital Business
Operational
Sources
Real-Time
Analytics
Oracle, SQL Server,
SAP, Hana, DB2
Dashboards, Real-Time
Applications
Pipelines Live Data Historical Data
Why MemSQL and Attunity?
• Live synchronization without impacting application performance
• Save time and effort reporting and analyzing directly on transactional data
• Cost effective with enterprise performance and operational capabilities
• Run anywhere spanning on-premises to cloud
• Augment Streaming ingest data with Enterprise ODS/Transactional context
Change Data Capture

Q&A – NEXT STEPS
MemSQL
go.memsql.com/oreilly-
data-pipelines
Attunity
bit.ly/cdcbook
Wikibon
wikibon.com/wikibons-2018-
big-data-analytics-trends-
forecast

Digital Business Transformation in the Streaming Era

More Related Content

What's hot

Similar to Digital Business Transformation in the Streaming Era

Recently uploaded

Digital Business Transformation in the Streaming Era