Downloaded 15 times
![About Me
2
Work [ed | s] @
Committer &
PPMC on
Father of 2
Co-Chair for
Apache Airflow](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-2-2048.jpg)
![84
Avro Schema Example
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-84-2048.jpg)
![85
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Avro Schema Example](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-85-2048.jpg)
![86
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
Avro Schema Example](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-86-2048.jpg)
![87
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
3 fields in the record: 1 required, 2
optional
Avro Schema Example](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-87-2048.jpg)
![88
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Data
x 1,000,000,000
Avro Schema Data File Example
Schema
Data
0.0001 %
99.999 %
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-88-2048.jpg)
![89
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro Schema Streaming Example
Schema
Data
99 %
1 %
Data](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-89-2048.jpg)
![90
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro Schema Streaming Example
Schema
Data
99 %
1 %
Data
OVERHEAD!!](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-90-2048.jpg)
![92
Schema
Registry
(Lambda)
Avro Schema Registry
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
register_schema
Message
Producer (P)](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-92-2048.jpg)
![96
Schema
Registry
(Lambda)
Message
Producer (P)
Data
Message
Consumer (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Avro Schema Registry](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-96-2048.jpg)
![97
Schema
Registry
(Lambda)
Message
Producer (P)
Message
Consumer (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Message Consumers
• download & cache the schema
• then decode the data
Avro Schema Registry](https://image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-97-2048.jpg)
![About Me
2
Work [ed | s] @
Committer &
PPMC on
Father of 2
Co-Chair for
Apache Airflow](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-2-2048.jpg)
![84
Avro Schema Example
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-84-2048.jpg)
![85
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Avro Schema Example](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-85-2048.jpg)
![86
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
Avro Schema Example](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-86-2048.jpg)
![87
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
complex type (record)
Schema name : User
3 fields in the record: 1 required, 2
optional
Avro Schema Example](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-87-2048.jpg)
![88
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Data
x 1,000,000,000
Avro Schema Data File Example
Schema
Data
0.0001 %
99.999 %
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data
Data](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-88-2048.jpg)
![89
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro Schema Streaming Example
Schema
Data
99 %
1 %
Data](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-89-2048.jpg)
![90
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Binary Data block
Avro Schema Streaming Example
Schema
Data
99 %
1 %
Data
OVERHEAD!!](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-90-2048.jpg)
![92
Schema
Registry
(Lambda)
Avro Schema Registry
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
register_schema
Message
Producer (P)](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-92-2048.jpg)
![96
Schema
Registry
(Lambda)
Message
Producer (P)
Data
Message
Consumer (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Avro Schema Registry](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-96-2048.jpg)
![97
Schema
Registry
(Lambda)
Message
Producer (P)
Message
Consumer (C)
getSchemaById (UUID)
{"namespace": "agari",
"type": "record",
"name": "User",
"fields": [
{"name": "name", "type": "string"},
{"name": "favorite_number", "type": ["int", "null"]},
{"name": "favorite_color", "type": ["string", "null"]}
]
}
Message Consumers
• download & cache the schema
• then decode the data
Avro Schema Registry](https://crownmelresort.com/image.slidesharecdn.com/cndp2017dataengconfv2-170427190233/75/Cloud-Native-Data-Pipelines-DataEngConf-SF-2017-97-2048.jpg)
Uploaded bySid Anand
Cloud Native Data Pipelines (DataEngConf SF 2017)
This document discusses cloud native data pipelines. It begins by introducing the speaker and their company, Agari, which applies trust models to email metadata to score messages. The document then discusses design goals for resilient data pipelines, including operability, correctness, timeliness and cost. It presents two use cases at Agari: batch message scoring and near real-time message scoring. For each use case, the pipeline architecture is shown including components like S3, SNS, SQS, ASGs, EMR and databases. The document discusses leveraging AWS services and tools like Airflow, Packer and Terraform to tackle issues like cost, timeliness, operability and correctness. It also introduces innovations like Apache Avro for
More Related Content
Similar to Cloud Native Data Pipelines (DataEngConf SF 2017)
Cloud Native Data Pipelines (DataEngConf SF 2017)
- 1. Cloud Native Data Pipelines 1 SidAnand (@r39132) DataEngConf SF 2017
- 2. About Me 2 Work [ed| s] @ Committer & PPMC on Father of 2 Co-Chair for Apache Airflow
- 3.
- 4. Agari : WhatWe Do 4
- 5. 5 Agari : WhatWe Do
- 6. 6 Agari : WhatWe Do
- 7. 7 Agari : WhatWe Do
- 8. 8 Agari : WhatWe Do
- 9. 9 Enterprise Customers email metadata apply trust models email md + trustscore Agari’s Previous EP Version Agari : What We Do Batch
- 10. 10 email metadata apply trust models email md + trustscore Agari’s Current EP VersionEnterprise Customers Agari : What We Do Near-real time Quarantine, Label, PassThrough
- 11. Data Pipelines BI vsPredictive 11
- 12.
- 13.
- 14.
- 15. BI Predictive Common Focusof this talk Data Pipelines 15 Web Servers OLTP DB Data Warehouse Repor6ng Tools Query Browsers ETL (batch) MySQL, Oracle, Cassandra Terradata, RedShi; BigQuery OLTP DB or cache ETL (batch or streaming) MySQL, Oracle, Cassandra, Redis Spark, Flink, Beam, Storm Web Servers Ranking (Search, News Feed), Recommender Products, Fraud DetecGon / PrevenGon Data Source
- 16. Motivation Cloud Native DataPipelines 16
- 17. Cloud Native DataPipelines 17 Big Data Companies like LinkedIn, Facebook, Twitter, & Google have large teams to manage their data pipelines Most start-ups run in the public cloud. Can they leverage aspects of the public cloud to build comparable pipelines?
- 18. Cloud Native DataPipelines 18 Cloud Native Techniques Open Source Technogies Data Pipelines seen in Big Data companies ~
- 19. Design Goals Desirable Qualitiesof a Resilient Data Pipeline 19
- 20. 20 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 21. 21 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • Data Integrity (no loss, etc…) • Expected data distributions • All output within time-bound SLAs • Minimize Operational Fatigue / Automate Everything • Fine-grained Monitoring & Alerting of Correctness & Timeliness SLAs • Quick Recoverability • Pay-as-you-go
- 22. Predictive Analytics @Agari Use Cases 22
- 23. Use Cases 23 Apply trustmodels (message scoring) batch + near real time Build trust models batch (Enterprise Protect)
- 24. Use Cases 24 Apply trustmodels (message scoring) batch + near real time Build trust models batch (Enterprise Protect) Focus of this talk
- 25. Use-Case : Message Scoring(batch) Batch Pipeline Architecture 25
- 26. Use-Case : MessageScoring 26 enterprise A enterprise B enterprise C S3 S3 uploads an Avro file every 15 minutes
- 27. Use-Case : MessageScoring 27 enterprise A enterprise B enterprise C S3 Airflow kicks of a Spark message scoring job every hour (EMR)
- 28. Use-Case : MessageScoring 28 enterprise A enterprise B enterprise C S3 Spark job writes scored messages and stats to another S3 bucket S3
- 29. Use-Case : MessageScoring 29 enterprise A enterprise B enterprise C S3 This triggers SNS/SQS messages events S3 SNS SQS
- 30. Use-Case : MessageScoring 30 enterprise A enterprise B enterprise C S3 An Autoscale Group (ASG) of Importers spins up when it detects SQS messages S3 SNS SQS Importers ASG
- 31. 31 enterprise A enterprise B enterpriseC S3 The importers rapidly ingest scored messages and aggregate statistics into the DB S3 SNS SQS Importers ASG DB Use-Case : Message Scoring
- 32. 32 enterprise A enterprise B enterpriseC S3 Users receive alerts of untrusted emails & can review them in the web app S3 SNS SQS Importers ASG DB Use-Case : Message Scoring
- 33. 33 enterprise A enterprise B enterpriseC S3 S3 SNS SQS Importers ASG DB Airflow manages the entire process Use-Case : Message Scoring
- 34. 34 Architectural Components Component RoleUses Salient Features Operability Model Data Lake • All data stored in S3 • All processing uses S3 Scalable, Available, Performant Serverless Messaging • Reliable, Transactional, Pub/Sub Scalable, Available, Performant Serverless ASG General Processing • Used for importing, data cleansing, business logic Scalable, Available, Performant Managed Data Science Processing • Aggregation • Model Building • Scoring Nice programming model at the cost of debugging complexity We Operate Workflow Engine • Coordinates all Spark Jobs & complex flows Lightweight, DAGs as Code, Steep learning curve We Operate DB Persistence for WebApp • Holds subset of data needed for Web App Rails + Postgres ‘nuff said We Operate S3 SNS SQS
- 35. Tackling Cost &Timeliness Leveraging the AWS Cloud 35
- 36. Tackling Cost 36 Between DailyRuns During Daily Runs When running daily, for 23 hours of a day, we didn’t pay for instances in the ASG or EMR
- 37. Tackling Cost 37 Between HourlyRuns During Hourly Runs When running daily, for 23 hours of a day, we didn’t pay for instances in the ASG or EMR This does not help when runs are hourly since AWS charges at an hourly rate for EC2 instances!
- 38. Tackling Timeliness Auto ScalingGroup (ASG) 38
- 39. ASG - Overview 39 Whatis it? A means to automatically scale out/in clusters to handle variable load/traffic A means to keep a cluster/service of a fixed size always up
- 40. ASG - DataPipeline 40 importer importer importer importer Importer ASG scaleout/in SQS DB
- 41. 41 Sent CPU ACKd/Recvd CPU-based auto-scaling is goodat scaling in/out to keep the average CPU constant ASG : CPU-based
- 42. ASG : CPU-based 42 Sent CPU Recv Premature Scale-in PrematureScale-in: • The CPU drops to noise-levels before all messages are consumed • This causes scale in to occur while the last few messages are still being committed
- 43. 43 Scale-out: When VisibleMessages > 0 (a.k.a. when queue depth > 0) Scale-in: When Invisible Messages = 0 (a.k.a. when the last in-flight message is ACK’d) This causes the ASG to grow This causes the ASG to shrink ASG : Queue-based
- 44. Auto Scaling Groups Build& Deploy 44
- 45. ASG - Build& Deploy 45 Component Role Details Spins up Cloud Resources • Spins up SQS, Kinesis, EC2, ASG, ELB, etc.. and associate them using Terraform • A better version of Chef & Puppet • Sets up an EC2 instance • Agentless, idempotent, & declarative tool to set up EC2 instances, by installing & configuring packages, and more • Spins up an EC2 instance for the purposes of building an AMI! • Can be used with Ansible & Terraform to bake AMIs & Launch Auto-Scaling Groups
- 46. ASG - Build& Deploy 46 EC2 Step 1 : Packer spins up a temporary EC2 node - a blank canvas!
- 47. EC2 ASG - Build& Deploy 47 EC2 Step 1 : Packer spins up a temporary EC2 node - a blank canvas! Step 2 : Packer runs an Ansible role against the EC2 node to set it up.
- 48. EC2 ASG - Build& Deploy 48 EC2 Step 2 : Packer runs an Ansible role against the EC2 node to set it up. Step 3 : Snapshots the machine & register the AMI.EC2 Step 1 : Packer spins up a temporary EC2 node - a blank canvas!
- 49. EC2 ASG - Build& Deploy 49 EC2 Step 2 : Packer runs an Ansible role against the EC2 node to set it up. Step 3 : Snapshots the machine & register the AMI.EC2 Step 4 : Terminates the EC2 instance! Step 1 : Packer spins up a temporary EC2 node - a blank canvas!
- 50. EC2 ASG - Build& Deploy 50 EC2 Step 2 : Packer runs an Ansible role against the EC2 node to set it up. Step 3 : Snapshots the machine & register the AMI.EC2 Step 4 : Terminates the EC2 instance! Step 5 : Using the AMI, Terraform spins up an auto-scaled compute cluster (ASG) Step 1 : Packer spins up a temporary EC2 node - a blank canvas! ASG
- 51. 51 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost • ASG • EMR Spark Daily • ASG • EMR Spark Hourly ASG • No Cost Savings
- 52.
- 53. 53 A simple wayto author, configure, manage workflows Provides visual insight into the state & performance of workflow runs Integrates with our alerting and monitoring tools Tackling Operability : Requirements
- 54. Apache Airflow Workflow Automation& Scheduling 54
- 55. 55 Airflow: Author DAGsin Python! No need to bundle many config files! Apache Airflow - Authoring DAGs
- 56. 56 Airflow: Visualizing aDAG Apache Airflow - Authoring DAGs
- 57. 57 Airflow: It’s easyto manage multiple DAGs Apache Airflow - Managing DAGs
- 58. Apache Airflow -Perf. Insights 58 Airflow: Gantt chart view reveals the slowest tasks for a run!
- 59. 59 Apache Airflow -Perf. Insights Airflow: Task Duration chart view show task completion time trends!
- 60. 60 Airflow: …And easyto integrate with Ops tools! Apache Airflow - Alerting
- 61. 61 Apache Airflow -Correctness
- 62. 62 Desirable Qualities ofa Resilient Data Pipeline OperabilityCorrectness Timeliness Cost
- 63. Use-Case : Message Scoring(near-real time) NRT Pipeline Architecture 63
- 64. Use-Case : MessageScoring 64 enterprise A enterprise B enterprise C Kinesis batch put every second K
- 65. Use-Case : MessageScoring 65 enterprise A enterprise B enterprise C K As ASG of scorers is scaled up to one process per core per kinesis shard Scorers ASG
- 66. Use-Case : MessageScoring 66 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Scorers apply the trust model and send scored messages downstream
- 67. Use-Case : MessageScoring 67 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG As ASG of importers is scaled up to rapidly import messages DB
- 68. Use-Case : MessageScoring 68 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG
- 69. Use-Case : MessageScoring 69 enterprise A enterprise B enterprise C K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG Quarantine Email
- 70. 70 Stream Processing Architecture ComponentRole Details Pros Operability Model Data Lake • All data stored in S3 via Kinesis Firehose Scalable, Available, Performant, Serverless Serverless Kinesis Messaging • Streaming transport modeled on Kafka Scalable, Available, Serverless Serverless General Processing • ASG Replacement except for Rails Apps Scalable, Available, Serverless Serverless ASG General Processing • Used for importing, data cleansing, business logic Scalable, Available, Managed Managed Data Science Processing • Model Building We Operate Workflow Engine • Nightly model builds + some classic Ops cron workloads Lightweight, DAGs as Code We Operate DB Persistence for WebApp • Holds smaller subset of data needed for Web App Rails + Postgres ‘nuff said We Operate Persistence for WebApp • Aggregation + Search moved from DB to ES • Model Building queries moved to Elasticache Redis Faster. more accurate for aggregates, frees up headroom for DB (polyglot persistence) Managed S3
- 71.
- 72. Apache Avro What isAvro? 72
- 73. 73 What is Avro? Avrois a self-describing serialization format that supports primitive data types : int, long, boolean, float, string, bytes, etc… complex data types : records, arrays, unions, maps, enums, etc… many language bindings : Java, Scala, Python, Ruby, etc…
- 74. 74 What is Avro? Avrois a self-describing serialization format that supports primitive data types : int, long, boolean, float, string, bytes, etc… complex data types : records, arrays, unions, maps, enums, etc… many language bindings : Java, Scala, Python, Ruby, etc… The most common format for storing structured Big Data at rest in HDFS, S3, Google Cloud Storage, etc… Supports Schema Evolution!
- 75. Apache Avro Why isit useful? 75
- 76. 76 Why is AvroUseful? Agari is an IoT company! Agari Sensors, deployed at customer sites, stream data to Agari’s Cloud SAAS Data is sent via Kinesis! enterprise A enterprise B enterprise C Kinesis Agari SAAS in AWS
- 77. 77 Why is AvroUseful? enterprise A : enterprise B : enterprise C : Kinesis v1 v2 v3 Agari is an IoT company! Agari Sensors, deployed at customer sites, stream data to Agari’s Cloud SAAS Data is sent via Kinesis! At any point in time, customers run different versions of the Agari Sensor Agari SAAS in AWS
- 78. 78 Why is AvroUseful? enterprise A : enterprise B : enterprise C : Kinesis v1 v2 v3 Agari is an IoT company! Agari Sensors, deployed at customer sites, stream data to Agari’s Cloud SAAS Data is sent via Kinesis! At any point in time, customers run different versions of the Agari Sensor These Sensors might send different format versions of the data! Agari SAAS in AWS
- 79. 79 Why is AvroUseful? enterprise A : enterprise B : enterprise C : Kinesis v1 v2 v3 Agari SAAS in AWS v4 Agari is an IoT company! Agari Sensors, deployed at customer sites, stream data to Agari’s Cloud SAAS Data is sent via Kinesis! At any point in time, customers run different versions of the Agari Sensor These Sensors might send different format versions of the data!
- 80. 80 Why is AvroUseful? enterprise A : enterprise B : enterprise C : v1 v2 v3 Avro allows Agari to seamlessly handle different IoT data format versions Agari SAAS in AWS Kinesis v4 datum_reader = DatumReader( writers_schema = writers_schema, readers_schema = readers_schema) Requirements: • Schemas are backward-compatible
- 81. 81 Why is AvroUseful? Agari SAAS in AWS S1 S2 S3 s3 Spark Avro Everywhere! Avro is so useful, we don’t just to communicate between our Sensors & our SAAS infrastructure We also use it as the common data-interchange format between all services (streaming & batch) within our AWS deployment
- 82. 82 Why is AvroUseful? Agari SAAS in AWS S1 S2 S3 s3 Spark Avro Everywhere! Good Language Bindings : Data Pipelines services are written in Java, Ruby, & Python
- 83.
- 84. 84 Avro Schema Example {"namespace":"agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] }
- 85. 85 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Avro Schema Example
- 86. 86 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Schema name : User Avro Schema Example
- 87. 87 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } complex type (record) Schema name : User 3 fields in the record: 1 required, 2 optional Avro Schema Example
- 88. 88 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Data x 1,000,000,000 Avro Schema Data File Example Schema Data 0.0001 % 99.999 % Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data Data
- 89. 89 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Binary Data block Avro Schema Streaming Example Schema Data 99 % 1 % Data
- 90. 90 {"namespace": "agari", "type": "record", "name":"User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Binary Data block Avro Schema Streaming Example Schema Data 99 % 1 % Data OVERHEAD!!
- 91.
- 92. 92 Schema Registry (Lambda) Avro Schema Registry {"namespace":"agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } register_schema Message Producer (P)
- 93. 93 Schema Registry (Lambda) register_schema returns aUUID Message Producer (P) Avro Schema Registry
- 94. 94 Schema Registry (Lambda) Message Producer sendsUUID + Message Producer (P) Data Message Consumer (C) Avro Schema Registry
- 95. 95 Schema Registry (Lambda) Message Producer (P) Data Message Consumer (C) getSchemaById(UUID) Avro Schema Registry
- 96. 96 Schema Registry (Lambda) Message Producer (P) Data Message Consumer (C) getSchemaById(UUID) {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Avro Schema Registry
- 97. 97 Schema Registry (Lambda) Message Producer (P) Message Consumer (C) getSchemaById(UUID) {"namespace": "agari", "type": "record", "name": "User", "fields": [ {"name": "name", "type": "string"}, {"name": "favorite_number", "type": ["int", "null"]}, {"name": "favorite_color", "type": ["string", "null"]} ] } Message Consumers • download & cache the schema • then decode the data Avro Schema Registry
- 98. 98 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG SR SR SR Avro Schema Registry
- 99. 99 enterprise A enterprise B enterpriseC K Scorers ASG Kinesis Importers ASG Imported messages are also consumed by the alerter DB K Alerters ASG SR SR SR Avro Schema Registry
- 100. Acknowledgments 100 • Vidur Apparao •Stephen Cattaneo • Jon Chase • Andrew Flury • William Forrester • Chris Haag • Chris Buchanan • Neil Chapin • Wil Collins • Don Spencer • Scot Kennedy • Natia Chachkhiani • Patrick Cockwell • Kevin Mandich • Gabriel Ortiz • Jacob Rideout • Josh Yang • Julian Mehnle • Gabriel Poon • Spencer Sun • Nathan Bryant None of this work would be possible without the essential contributions of the team below
- 101.