Azure Data Factory ETL Patterns in the Cloud

ETL Patterns in the Cloud with
Azure Data Factory
Mark Kromer
Senior Program Manager
Microsoft Azure Data Management
@kromerbigdata

ETL Patterns in the Cloud
Important factors for success
1. What is ETL?
• More than Extract, Transform, Load
• Scheduling, Monitoring, Maintenance, Source Control, CI/CD, Operationalize
2. Platform as a Service (ADF) vs. Infrastructure as a Service (IaaS/SSIS)
• Self-managed vs. Provider-Managed
3. ELT or ETL?
• Difference is primarily highly-parsed semantics
• However: In the cloud, common pattern == stage data in low-cost, inexpensive storage
4. Not typically performant to process data in-flight
• Particularly crossing boundaries (on-prem, vnets, data centers, regions)
5. Scale is very important in Cloud ETL
• Cloud projects assume elastic scale. ETL is not immune to this expectation.
6. Flexible Schema is very important in Cloud ETL
1. Assume “Big Data tenets” aka “data chaos”: Your data sources will change shape, size and
volume. Often!

Azure Data Factory
Workflow Pipelines/Control Flow

Built-in source
control support

Quickly get started with building data integration solutions. Avoid building same workflows
repeatedly. Simply instantiate a template. Improve developer productivity along with reducing
development time for repeat processes.
Use Templates to quickly get started

Secure data platform enabling Analytics and Insights on Microsoft 365.
Data access @ Scale
Dataset based access rather than
real time API based access
Granular Request/Consent
enabling Data Privacy
Row and column level scoping
with advanced filtering capability
Data Governance & Security
Control and visibility over your
data throughout its entire lifecycle
Microsoft Graph Data Connect

ADF Integration Runtime
Activity Dispatch/Monitor
Data Movement
SSIS Package Execution

Azure Data Factory Service
Cloud Apps, Svcs & DataOn Premises Apps & Data
UX & SDK

Azure (US West)
Public Internet Border
HP Inc (Global)
HP Prod Firewall Border
HP Hadoop Cluster
Integration Fabric On-prem
Data Factory
(Orchestration
micro-service)
443
Storage (Azure)
443
SQL Data
Warehouse
UAM Server
443
ADF Foo On-prem
“IR”
Customer 1
Customer 1 firewall border
Azure Data Factory “Integration Runtime” deployed on premises for
transformation and then moved to cloud

on premises
in Azure
Deployment via SSMS
Once connected, you can deploy
projects/packages to SSIS PaaS from your
local file system/SSIS on premises

Execution via SSMS
You can select some packages to execute
on SSIS PaaS

Monitoring via SSMS
on premises
in Azure
You can see package execution
error messages

Execute SSIS Packaged in ADF Pipeline

What is ADF Mapping Data Flow?
Transform Data, At Scale, in the Cloud,
Zero-Code
Cloud-first, scale-out ELT
Code-free dataflow pipelines
Serverless scale-out transformation
execution engine
Maximum Productivity for Data
Engineers
Does NOT require understanding of Spark /
Scala / Python / Java
Resilient Data Transformation Flows
Built for big data scenarios with
unstructured data requirements
Operationalize with Data Factory
scheduling, control flow and monitoring

Code-free Data Transformation At Scale
Does not require understanding of Spark, Big Data Execution
Engines, Clusters, Scala, Python …
Focus on building business logic and data transformation
Data cleansing
Aggregation
Data conversions
Data prep
Data exploration

ADF Data Flow Workstream
Stage Data in Azure
(ADLS, Blob, SQL
DB/DW)
Transform Data in
Visual Data Flow
Land Data in Azure
Staging Area (ADLS,
Blob, SQL DB/DW)

Build your logical data flows adding data
transformations in a guided experience

Microsoft Azure Data Factory Continues to Extend Data Flow
Library with a Rich Set of Transformations and Expression
Functions

Debug mode provides row-level context
and visible results in inspector pane

Interactive Expression Builder – Build data transform
expressions, not Spark code

Deep Monitoring Introspection of Data Transformations

Debug Data Flows with Data Preview and Data Sampling

MODEL & SERVE
Azure Analysis ServicesAzure SQL Data
Warehouse
Power BI
Modernize your enterprise data warehouse at scale
A Z U R E D A T A F A C T O R Y
On-premises data
Oracle, SQL, Teradata,
fileshares, SAP
Cloud data
Azure, AWS, GCP
SaaS data
Salesforce, Workday,
Dynamics
INGEST STORE PREP & TRAIN
Azure Data Factory Azure Blob Storage
Azure Databricks
Polybase
Microsoft Azure also supports other Big Data services like Azure HDInsight, Azure SQL Database and Azure Data Lake to allow customers to tailor the above architecture to meet their unique needs.
Orchestrate with Azure Data Factory

Lift your SQL Server Integration Services (SSIS) packages to Azure
On-Premise data sources
SQL DB Managed Instance
SQL Server
VNET
Azure Data Factory
SSIS Cloud ETL
SSIS Integration Runtime
Cloud data sources
Cloud
On-premises
Microsoft
SQL Server
Integration Services

Author, orchestrate and monitor with Azure Data Factory
Hybrid and Multi-Cloud Data Integration
Azure Data Factory
PaaS Data Integration
DATA SCIENCE
AND MACHINE
LEARNING
MODELS
ANALYTICAL
DASHBOARDS
USING POWER BI
DATA DRIVEN
APPLICATIONS
On-Prem SaaS Apps Public Cloud

Nightly ETL Data Flows - Codeless

Build Resilient Data Flows with Schema Drift
Handling of Flexible Schemas

Data Engineer derives columns using template expression
patterns based on name and type matching. No need to define
static field names.

Sink all incoming fields along with new
derived field

Slowly Changing Dimension Scenario

Data Lake Data Science Scenario

Azure Data Factory ETL Patterns in the Cloud

In this document

More Related Content

What's hot

Similar to Azure Data Factory ETL Patterns in the Cloud

More from Mark Kromer

Recently uploaded

Azure Data Factory ETL Patterns in the Cloud