Databricks, profile picture
Uploaded byDatabricks
PDF, PPTX2,943 views

Easy, scalable, fault tolerant stream processing with structured streaming - spark summit 2017

The document discusses structured streaming with Apache Spark, focusing on its ease of use, scalability, and fault tolerance for stream processing applications. It highlights key features such as data integration from various sources, checkpointing for fault tolerance, and advanced transformations to handle complex workloads and data types. The presentation also covers practical examples of implementing streaming queries, event-time aggregations, and stateful processing with benefits for real-time analytics and decision-making.

Software◦
Related topics:
Apache Spark•

Embed presentation

Download as PDF, PPTX
Easy, Scalable, Fault-tolerant Stream Processing with Structured Streaming Michael Armbrust - @michaelarmbrust Tathagata Das - @tathadas Spark Summit 2017 6th June, San Francisco
TEAM About Databricks Started Spark project (now Apache Spark) at UC Berkeley in 2009 PRODUCT Unified Analytics Platform MISSION Making Big Data Simple
building robust stream processing apps is hard
Complexities in stream processing COMPLEX DATA Diverse data formats (json, avro, binary, …) Data can be dirty, late, out-of-order COMPLEX SYSTEMS Diverse storage systems (Kafka, S3, Kinesis, RDBMS, …) System failures COMPLEX WORKLOADS Combining streaming with interactive queries Machine learning
Structured Streaming stream processing on Spark SQL engine fast, scalable, fault-tolerant rich, unified, high level APIs deal with complex data and complex workloads rich ecosystem of data sources integrate with many storage systems
you should not have to reason about streaming
you should write simple queries & Spark should continuously update the answer
Streaming word count Anatomy of a Streaming Query
Anatomy of a Streaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy($"value".cast("string")) .count() .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode(OutputMode.Complete()) .option("checkpointLocation", "…") .start() Source • Specify one or more locations to read data from • Built in support for Files/Kafka/Socket, pluggable. • Can include multiple sources of different types using union()
Anatomy of a Streaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode(OutputMode.Complete()) .option("checkpointLocation", "…") .start() Transformation • Using DataFrames, Datasets and/or SQL. • Catalyst figures out how to execute the transformation incrementally. • Internal processing always exactly-once.
DataFrames, Datasets, SQL input = spark.readStream .format("kafka") .option("subscribe", "topic") .load() result = input .select("device", "signal") .where("signal > 15") result.writeStream .format("parquet") .start("dest-path") Logical Plan Read from Kafka Project device, signal Filter signal > 15 Write to Kafka Spark automatically streamifies! Spark SQL converts batch-like query to a series of incremental execution plans operating on new batches of data Series of Incremental Execution Plans Kafka Source Optimized Operator codegen, off- heap, etc. Kafka Sink Optimized Physical Plan process newdata t = 1 t = 2 t = 3 process newdata process newdata
Anatomy of a Streaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode(OutputMode.Complete()) .option("checkpointLocation", "…") .start() Sink • Accepts the output of each batch. • When supported sinks are transactional and exactly once (Files). • Use foreach to execute arbitrary code.
Anatomy of a Streaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode("update") .option("checkpointLocation", "…") .start() Output mode – What's output • Complete – Output the whole answer every time • Update – Output changed rows • Append – Output new rows only Trigger – When to output • Specified as a time, eventually supports data size • No trigger means as fast as possible
Anatomy of a Streaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode("update") .option("checkpointLocation", "…") .start() Checkpoint • Tracks the progress of a query in persistent storage • Can be used to restart the query if there is a failure.
Fault-tolerance with Checkpointing Checkpointing – tracks progress (offsets) of consuming data from the source and intermediate state. Offsets and metadata saved as JSON Can resume after changing your streaming transformations end-to-end exactly-once guarantees process newdata t = 1 t = 2 t = 3 process newdata process newdata write ahead log
Complex Streaming ETL
Traditional ETL Raw, dirty, un/semi-structured is data dumped as files Periodic jobs run every few hours to convert raw data to structured data ready for further analytics 17 file dump seconds hours table 10101010
Traditional ETL Hours of delay before taking decisions on latest data Unacceptable when time is of essence [intrusion detection, anomaly detection, etc.] file dump seconds hours table 10101010
Streaming ETL w/ Structured Streaming Structured Streaming enables raw data to be available as structured data as soon as possible 19 seconds table 10101010
Streaming ETL w/ Structured Streaming Example Json data being received in Kafka Parse nested json and flatten it Store in structured Parquet table Get end-to-end failure guarantees val rawData = spark.readStream .format("kafka") .option("kafka.boostrap.servers",...) .option("subscribe", "topic") .load() val parsedData = rawData .selectExpr("cast (value as string) as json")) .select(from_json("json", schema).as("data")) .select("data.*") val query = parsedData.writeStream .option("checkpointLocation", "/checkpoint") .partitionBy("date") .format("parquet") .start("/parquetTable")
Reading from Kafka Specify options to configure How? kafka.boostrap.servers => broker1,broker2 What? subscribe => topic1,topic2,topic3 // fixed list of topics subscribePattern => topic* // dynamic list of topics assign => {"topicA":[0,1] } // specific partitions Where? startingOffsets => latest(default) / earliest / {"topicA":{"0":23,"1":345} } val rawData = spark.readStream .format("kafka") .option("kafka.boostrap.servers",...) .option("subscribe", "topic") .load()
Reading from Kafka val rawData = spark.readStream .format("kafka") .option("kafka.boostrap.servers",...) .option("subscribe", "topic") .load() rawData dataframe has the following columns key value topic partition offset timestamp [binary] [binary] "topicA" 0 345 1486087873 [binary] [binary] "topicB" 3 2890 1486086721
Transforming Data Cast binary value to string Name it column json val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*")
Transforming Data Cast binary value to string Name it column json Parse json string and expand into nested columns, name it data val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*") json { "timestamp": 1486087873, "device": "devA", …} { "timestamp": 1486082418, "device": "devX", …} data (nested) timestamp device … 1486087873 devA … 1486086721 devX … from_json("json") as "data"
Transforming Data Cast binary value to string Name it column json Parse json string and expand into nested columns, name it data Flatten the nested columns val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*") data (nested) timestamp device … 1486087873 devA … 1486086721 devX … timestamp device … 1486087873 devA … 1486086721 devX … select("data.*") (not nested)
Transforming Data Cast binary value to string Name it column json Parse json string and expand into nested columns, name it data Flatten the nested columns val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*") powerful built-in APIs to perform complex data transformations from_json, to_json, explode, ... 100s of functions (see our blog post)
Writing to Save parsed data as Parquet table in the given path Partition files by date so that future queries on time slices of data is fast e.g. query on last 48 hours of data val query = parsedData.writeStream .option("checkpointLocation", ...) .partitionBy("date") .format("parquet") .start("/parquetTable")
Checkpointing Enable checkpointing by setting the checkpoint location to save offset logs start actually starts a continuous running StreamingQuery in the Spark cluster val query = parsedData.writeStream .option("checkpointLocation", ...) .format("parquet") .partitionBy("date") .start("/parquetTable/")
Streaming Query query is a handle to the continuously running StreamingQuery Used to monitor and manage the execution val query = parsedData.writeStream .option("checkpointLocation", ...) .format("parquet") .partitionBy("date") .start("/parquetTable")/") process newdata t = 1 t = 2 t = 3 process newdata process newdata StreamingQuery
Data Consistency on Ad-hoc Queries Data available for complex, ad-hoc analytics within seconds Parquet table is updated atomically, ensures prefix integrity Even if distributed, ad-hoc queries will see either all updates from streaming query or none, read more in our blog https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html complex, ad-hoc queries on latest data seconds!
More Kafka Support [Spark 2.2] Write out to Kafka Dataframe must have binary fields named key and value Direct, interactive and batch queries on Kafka Makes Kafka even more powerful as a storage platform! result.writeStream .format("kafka") .option("topic", "output") .start() val df = spark .read // not readStream .format("kafka") .option("subscribe", "topic") .load() df.registerTempTable("topicData") spark.sql("select value from topicData")
Amazon Kinesis [Databricks Runtime 3.0] Configure with options (similar to Kafka) How? region => us-west-2 / us-east-1 / ... awsAccessKey (optional) => AKIA... awsSecretKey (optional) => ... What? streamName => name-of-the-stream Where? initialPosition => latest(default) / earliest / trim_horizon spark.readStream .format("kinesis") .option("streamName", "myStream") .option("region", "us-west-2") .option("awsAccessKey", ...) .option("awsSecretKey", ...) .load()
Working With Time
Event Time Many use cases require aggregate statistics by event time E.g. what's the #errors in each system in the 1 hour windows? Many challenges Extracting event time from data, handling late, out-of-order data DStream APIs were insufficient for event-time stuff
Event time Aggregations Windowing is just another type of grouping in Struct. Streaming number of records every hour Support UDAFs! parsedData .groupBy(window("timestamp","1 hour")) .count() parsedData .groupBy( "device", window("timestamp","10 mins")) .avg("signal") avg signal strength of each device every 10 mins
Stateful Processing for Aggregations Aggregates has to be saved as distributed state between triggers Each trigger reads previous state and writes updated state State stored in memory, backed by write ahead log in HDFS/S3 Fault-tolerant, exactly-once guarantee! process newdata t = 1 sink src t = 2 process newdata sink src t = 3 process newdata sink src state state write ahead log state updates are written to log for checkpointing state
Automatically handles Late Data 12:00 - 13:00 1 12:00 - 13:00 3 13:00 - 14:00 1 12:00 - 13:00 3 13:00 - 14:00 2 14:00 - 15:00 5 12:00 - 13:00 5 13:00 - 14:00 2 14:00 - 15:00 5 15:00 - 16:00 4 12:00 - 13:00 3 13:00 - 14:00 2 14:00 - 15:00 6 15:00 - 16:00 4 16:00 - 17:00 3 13:00 14:00 15:00 16:00 17:00Keeping state allows late data to update counts of old windows red = state updated with late data But size of the state increases indefinitely if old windows are not dropped
Watermarking Watermark - moving threshold of how late data is expected to be and when to drop old state Trails behind max seen event time Trailing gap is configurable event time max event time watermark data older than watermark not expected 12:30 PM 12:20 PM trailing gap of 10 mins
Watermarking Data newer than watermark may be late, but allowed to aggregate Data older than watermark is "too late" and dropped Windows older than watermark automatically deleted to limit the amount of intermediate state max event time event time watermark late data allowed to aggregate data too late, dropped
Watermarking max event time event time watermark allowed lateness of 10 mins parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() late data allowed to aggregate data too late, dropped Useful only in stateful operations (streaming aggs, dropDuplicates, mapGroupsWithState, ...) Ignored in non-stateful streaming queries and batch queries
Watermarking data too late, ignored in counts, state dropped Processing Time12:00 12:05 12:10 12:15 12:10 12:15 12:20 12:07 12:13 12:08 EventTime 12:15 12:18 12:04 watermark updated to 12:14 - 10m = 12:04 for next trigger, state < 12:04 deleted data is late, but considered in counts parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() system tracks max observed event time 12:08 wm = 12:04 10min 12:14 More details in my blog post
Clean separation of concerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics Processing Details separated from
Clean separation of concerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics How to group data by time? (same for batch & streaming) Processing Details
Clean separation of concerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics How to group data by time? (same for batch & streaming) Processing Details How late can data be?
Clean separation of concerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics How to group data by time? (same for batch & streaming) Processing Details How late can data be? How often to emit updates?
Arbitrary Stateful Operations [Spark 2.2] mapGroupsWithState allows any user-defined stateful function to a user-defined state Direct support for per-key timeouts in event-time or processing-time Supports Scala and Java 46 ds.groupByKey(_.id) .mapGroupsWithState (timeoutConf) (mappingWithStateFunc) def mappingWithStateFunc( key: K, values: Iterator[V], state: GroupState[S]): U = { // update or remove state // set timeouts // return mapped value }
Other interesting operations Streaming Deduplication Watermarks to limit state Stream-batch Joins Stream-stream Joins Can use mapGroupsWithState Direct support oming soon! val batchData = spark.read .format("parquet") .load("/additional-data") parsedData.join(batchData, "device") parsedData.dropDuplicates("eventId")
Building Complex Continuous Apps
Metric Processing @ Dashboards Analyze trends in usage as they occur Alerts Notify engineers of critical issues Ad-hoc Analysis Diagnose issues when they occur ETL Clean, normalize and store historical data Events generated by user actions (logins, clicks, spark job updates)
Metric Processing @ Dashboards Alerts Ad-hoc Analysis ETL Difficult with only streaming frameworks Limited retention in streaming storage Inefficient for ad-hoc queries Hard for novice users (limited or no SQL support)
Metric Processing @ = Metrics Filter ETL Dashboards Ad-hoc Analysis Alerts
Read from rawLogs = spark.readStream .format("kafka") .option("kafka.bootstrap.servers", ...) .option("subscribe", "rawLogs") .load() augmentedLogs = rawLogs .withColumn("msg", from_json($"value".cast("string"), schema)) .select("timestamp", "msg.*") .join(table("customers"), ["customer_id"]) DataFrames can be reused for multiple streams Can build libraries of useful DataFrames and share code between applications JSON ETL
Write to augmented .repartition(1) .writeStream .format("parquet") .option("path", "/data/metrics") .trigger("1 minute") .start() Store augmented stream as efficient columnar data for later processing Latency: ~1 minute Buffer data and write one large file every minute for efficient reads ETL
Dashboards logins = spark.readStream.parquet("/data/metrics") .where("metric = 'login'") .groupBy(window("timestamp", "1 minute")) .count() display(logins) // Visualize in Databricks notebooks Always up-to-date visualizations of important business trends Latency: ~1 minute to hours (configurable) Dashboards
Filter and write to filteredLogs = augmentedLogs .where("eventType = 'clusterHeartbeat'") .selectExpr("to_json(struct("*")) as value") filteredLogs.writeStream .format("kafka") .option("kafka.bootstrap.servers", ...) .option("topic", "clusterHeartbeats") .start() Forward filtered and augmented events back to Kafka Latency: ~100ms average Filter to_json() to convert columns back into json string, and then save as different Kafka topic
Simple Alerts sparkErrors .as[ClusterHeartBeat] .filter(_.load > 99) .writeStream .foreach(new PagerdutySink(credentials)) E.g. Alert when Spark cluster load > threshold Latency: ~100 ms Alerts Notify PagerDuty
Complex Alerts sparkErrors .as[ClusterHeartBeat] .groupBy(_.id) .flatMapGroupsWithState(Update, ProcessingTimeTimeout("1 minute")) { (id: Int, events: Iterator[ClusterHeartBeat], state: GroupState[ClusterState]) => ... // check if cluster non-responsive for a while } E.g. Monitor health of Spark clusters using custom stateful logic Latency: ~10 seconds Alerts React if no heartbeat from cluster for 1 min
Ad-hoc Analysis SELECT * FROM parquet.`/data/metrics` WHERE level IN ('WARN', 'ERROR') AND customer = "…" AND timestamp < now() – INTERVAL 1 HOUR Trouble shoot problems as they occur with latest information Latency: ~1 minute Ad-hoc Analysis will read latest data when query executed
Metric Processing @ = Metrics Filter ETL Dashboards Ad-hoc Analysis Alerts14+ billion records / hour with 10 nodes meet diverse latency requirements as efficiently as possible
More Info Structured Streaming Programming Guide http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html Databricks blog posts for more focused discussions https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html https://databricks.com/blog/2017/01/19/real-time-streaming-etl-structured-streaming-apache-spark-2-1.html https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html https://databricks.com/blog/2017/04/26/processing-data-in-apache-kafka-with-structured-streaming-in-apache-spark-2-2.html https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html and more to come, stay tuned!!
UNIFIED ANALYTICS PLATFORM Try Apache Spark in Databricks! • Collaborative cloud environment • Free version (community edition) DATABRICKS RUNTIME 3.0 • Apache Spark - optimized for the cloud • Caching and optimization layer - DBIO • Enterprise security - DBES Try for free today databricks.com

More Related Content

PDF
Structured Streaming for Columnar Data Warehouses with Jack Gudenkauf
byDatabricks
 
PDF
Easy, scalable, fault tolerant stream processing with structured streaming - ...
byDatabricks
 
PDF
Demystifying DataFrame and Dataset
byKazuaki Ishizaki
 
PDF
Productionizing your Streaming Jobs
byDatabricks
 
PDF
Making Structured Streaming Ready for Production
byDatabricks
 
PDF
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
byDatabricks
 
PDF
Apache Spark RDDs
byDean Chen
 
PDF
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
byInfluxData
 
Structured Streaming for Columnar Data Warehouses with Jack Gudenkauf
byDatabricks
 
Easy, scalable, fault tolerant stream processing with structured streaming - ...
byDatabricks
 
Demystifying DataFrame and Dataset
byKazuaki Ishizaki
 
Productionizing your Streaming Jobs
byDatabricks
 
Making Structured Streaming Ready for Production
byDatabricks
 
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
byDatabricks
 
Apache Spark RDDs
byDean Chen
 
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
byInfluxData
 

What's hot

PPTX
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
byspark-project
 
PDF
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
bySpark Summit
 
PDF
Sparkcamp @ Strata CA: Intro to Apache Spark with Hands-on Tutorials
byDatabricks
 
PDF
Deep Dive: Memory Management in Apache Spark
byDatabricks
 
PPTX
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
byChris Fregly
 
PPTX
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
byIlya Ganelin
 
PDF
Introducing DataFrames in Spark for Large Scale Data Science
byDatabricks
 
PDF
Spark Summit EU 2015: Spark DataFrames: Simple and Fast Analysis of Structure...
byDatabricks
 
PPTX
Spark etl
byImran Rashid
 
PDF
Data Source API in Spark
byDatabricks
 
PDF
DataEngConf SF16 - Spark SQL Workshop
byHakka Labs
 
PPTX
Tuning and Debugging in Apache Spark
byPatrick Wendell
 
PPTX
Building a modern Application with DataFrames
bySpark Summit
 
PDF
Automated Spark Deployment With Declarative Infrastructure
bySpark Summit
 
PPTX
Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...
byDataWorks Summit
 
PDF
Simplifying Big Data Analytics with Apache Spark
byDatabricks
 
PDF
Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)
bySpark Summit
 
PPTX
Frustration-Reduced PySpark: Data engineering with DataFrames
byIlya Ganelin
 
PDF
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 
PDF
What's new with Apache Spark's Structured Streaming?
byMiklos Christine
 
Deep Dive with Spark Streaming - Tathagata Das - Spark Meetup 2013-06-17
byspark-project
 
Deep Dive into Project Tungsten: Bringing Spark Closer to Bare Metal-(Josh Ro...
bySpark Summit
 
Sparkcamp @ Strata CA: Intro to Apache Spark with Hands-on Tutorials
byDatabricks
 
Deep Dive: Memory Management in Apache Spark
byDatabricks
 
Advanced Apache Spark Meetup Spark SQL + DataFrames + Catalyst Optimizer + Da...
byChris Fregly
 
Frustration-Reduced Spark: DataFrames and the Spark Time-Series Library
byIlya Ganelin
 
Introducing DataFrames in Spark for Large Scale Data Science
byDatabricks
 
Spark Summit EU 2015: Spark DataFrames: Simple and Fast Analysis of Structure...
byDatabricks
 
Spark etl
byImran Rashid
 
Data Source API in Spark
byDatabricks
 
DataEngConf SF16 - Spark SQL Workshop
byHakka Labs
 
Tuning and Debugging in Apache Spark
byPatrick Wendell
 
Building a modern Application with DataFrames
bySpark Summit
 
Automated Spark Deployment With Declarative Infrastructure
bySpark Summit
 
Modus operandi of Spark Streaming - Recipes for Running your Streaming Applic...
byDataWorks Summit
 
Simplifying Big Data Analytics with Apache Spark
byDatabricks
 
Exactly-Once Streaming from Kafka-(Cody Koeninger, Kixer)
bySpark Summit
 
Frustration-Reduced PySpark: Data engineering with DataFrames
byIlya Ganelin
 
Spark SQL Deep Dive @ Melbourne Spark Meetup
byDatabricks
 
What's new with Apache Spark's Structured Streaming?
byMiklos Christine
 

Viewers also liked

PDF
Building Continuous Application with Structured Streaming and Real-Time Data ...
byDatabricks
 
PDF
Introduction to Structured Streaming
bydatamantra
 
PPTX
Structured Streaming Using Spark 2.1
bySigmoid
 
PDF
A Deep Dive into Structured Streaming in Apache Spark
byDatabricks
 
ODP
Introduction to Structured Streaming
byKnoldus Inc.
 
PPTX
A Deep Dive into Structured Streaming: Apache Spark Meetup at Bloomberg 2016
byDatabricks
 
PDF
Introduction to Structured streaming
bydatamantra
 
PDF
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
byDatabricks
 
PDF
Deep dive into stateful stream processing in structured streaming by Tathaga...
byDatabricks
 
Building Continuous Application with Structured Streaming and Real-Time Data ...
byDatabricks
 
Introduction to Structured Streaming
bydatamantra
 
Structured Streaming Using Spark 2.1
bySigmoid
 
A Deep Dive into Structured Streaming in Apache Spark
byDatabricks
 
Introduction to Structured Streaming
byKnoldus Inc.
 
A Deep Dive into Structured Streaming: Apache Spark Meetup at Bloomberg 2016
byDatabricks
 
Introduction to Structured streaming
bydatamantra
 
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
byDatabricks
 
Deep dive into stateful stream processing in structured streaming by Tathaga...
byDatabricks
 

Similar to Easy, scalable, fault tolerant stream processing with structured streaming - spark summit 2017

PPTX
Kafka Summit NYC 2017 - Easy, Scalable, Fault-tolerant Stream Processing with...
byconfluent
 
PDF
Easy, scalable, fault tolerant stream processing with structured streaming - ...
byDatabricks
 
PDF
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
byDatabricks
 
PDF
Writing Continuous Applications with Structured Streaming in PySpark
byDatabricks
 
PDF
Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark.pdf
bynilanjan172nsvian
 
PDF
Writing Continuous Applications with Structured Streaming PySpark API
byDatabricks
 
PDF
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
byDatabricks
 
PDF
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
byDatabricks
 
PDF
Easy, Scalable, Fault-tolerant stream processing with Structured Streaming in...
byDataWorks Summit
 
PDF
Productizing Structured Streaming Jobs
byDatabricks
 
PPTX
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
byMicrosoft Tech Community
 
PPTX
Spark Structured Streaming
byRevin Chalil
 
PDF
Designing Structured Streaming Pipelines—How to Architect Things Right
byDatabricks
 
PDF
Continuous Application with Structured Streaming 2.0
byDatabricks
 
PPT
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
byDataWorks Summit
 
PDF
Getting Started With Spark Structured Streaming With Dustin Vannoy | Current ...
byHostedbyConfluent
 
PPTX
Getting Started with Spark Structured Streaming - Current 22
byDustin Vannoy
 
PPTX
Meetup spark structured streaming
byJosé Carlos García Serrano
 
PPTX
Stream, stream, stream: Different streaming methods with Spark and Kafka
byItai Yaffe
 
PDF
Structured Streaming in Spark
byDigital Vidya
 
Kafka Summit NYC 2017 - Easy, Scalable, Fault-tolerant Stream Processing with...
byconfluent
 
Easy, scalable, fault tolerant stream processing with structured streaming - ...
byDatabricks
 
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
byDatabricks
 
Writing Continuous Applications with Structured Streaming in PySpark
byDatabricks
 
Real-Time Data Pipelines Made Easy with Structured Streaming in Apache Spark.pdf
bynilanjan172nsvian
 
Writing Continuous Applications with Structured Streaming PySpark API
byDatabricks
 
A Deep Dive into Stateful Stream Processing in Structured Streaming with Tath...
byDatabricks
 
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
byDatabricks
 
Easy, Scalable, Fault-tolerant stream processing with Structured Streaming in...
byDataWorks Summit
 
Productizing Structured Streaming Jobs
byDatabricks
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
byMicrosoft Tech Community
 
Spark Structured Streaming
byRevin Chalil
 
Designing Structured Streaming Pipelines—How to Architect Things Right
byDatabricks
 
Continuous Application with Structured Streaming 2.0
byDatabricks
 
Stream, Stream, Stream: Different Streaming Methods with Spark and Kafka
byDataWorks Summit
 
Getting Started With Spark Structured Streaming With Dustin Vannoy | Current ...
byHostedbyConfluent
 
Getting Started with Spark Structured Streaming - Current 22
byDustin Vannoy
 
Meetup spark structured streaming
byJosé Carlos García Serrano
 
Stream, stream, stream: Different streaming methods with Spark and Kafka
byItai Yaffe
 
Structured Streaming in Spark
byDigital Vidya
 

More from Databricks

PPTX
DW Migration Webinar-March 2022.pptx
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 2
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 4
byDatabricks
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
PDF
Learn to Use Databricks for Data Science
byDatabricks
 
PDF
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
PDF
Sawtooth Windows for Feature Aggregations
byDatabricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
PDF
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 
DW Migration Webinar-March 2022.pptx
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
Data Lakehouse Symposium | Day 2
byDatabricks
 
Data Lakehouse Symposium | Day 4
byDatabricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
Learn to Use Databricks for Data Science
byDatabricks
 
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
Sawtooth Windows for Feature Aggregations
byDatabricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 

Recently uploaded

PDF
Comprehensive Windows protection with SiyanoAV Total Security — fast, smart, ...
byyogeshchauhanoffice
 
PDF
2025_11_19 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
PDF
vMix Overview & Key Features Easily Produce, Record and Live Stream
byIGM Soraing
 
PDF
DSD-INT 2025 Thermal and chemical plumes of sea cooling water from PEM platfo...
byDeltares
 
PDF
DSD-INT 2025 3D Modeling of Shallow Water Dynamics Using Delft3D FM - Martins
byDeltares
 
PDF
Hyperon Chain – Smart Contract Security Audit Report by EtherAuthority
byEtherAuthority1
 
PDF
AI/ML Infra Meetup | Open Source Michelangelo: Uber's Predictive to Generativ...
byAlluxio, Inc.
 
PDF
Speculative Automated Refactoring of Imperative Deep Learning Programs to Gra...
byRaffi Khatchadourian
 
PPTX
Working of Minimax Algorithm with Alpha-Beta Pruning.pptx
byMuhammadHassanArshad2
 
PDF
AI/ML Infra Meetup | SkyPilot: Open-source System to Scale AI across Clusters...
byAlluxio, Inc.
 
PDF
SWEBOK: the software engineering body of knowledge (SC25 Workshop Research So...
byHironori Washizaki
 
PDF
AOX Apps - Mobile App Development Company New York
byAOX APPS
 
PDF
NYC ACE 12-Nov-2025-Combined Presentation.pdf
byAUGNYC
 
PDF
DSD-INT 2025 Exploring different domain decomposition approaches for enhanced...
byDeltares
 
PPTX
Communicating Software Architecture using Arc42 v1.1
byAshraf Fouad
 
PPTX
Top Benefits of Hiring a Mobile App Development Company in India
byDevherds software solution
 
PDF
A Complete Guide to Salesforce for Education.pdf
byVALiNTRY360
 
PDF
How to Make Your AI Reliable: Comprehensive Guide to Testing AI Applications ...
byQATestLab Quality is a rule
 
PDF
From Chatbot to AI Agent: A Practical Comparison
byShiv Technolabs Pvt. Ltd.
 
PDF
SCORM Cloud: The 5 categories of content distribution
byRustici Software
 
Comprehensive Windows protection with SiyanoAV Total Security — fast, smart, ...
byyogeshchauhanoffice
 
2025_11_19 - OpenMetadata Community Meeting.pdf
byOpenMetadata
 
vMix Overview & Key Features Easily Produce, Record and Live Stream
byIGM Soraing
 
DSD-INT 2025 Thermal and chemical plumes of sea cooling water from PEM platfo...
byDeltares
 
DSD-INT 2025 3D Modeling of Shallow Water Dynamics Using Delft3D FM - Martins
byDeltares
 
Hyperon Chain – Smart Contract Security Audit Report by EtherAuthority
byEtherAuthority1
 
AI/ML Infra Meetup | Open Source Michelangelo: Uber's Predictive to Generativ...
byAlluxio, Inc.
 
Speculative Automated Refactoring of Imperative Deep Learning Programs to Gra...
byRaffi Khatchadourian
 
Working of Minimax Algorithm with Alpha-Beta Pruning.pptx
byMuhammadHassanArshad2
 
AI/ML Infra Meetup | SkyPilot: Open-source System to Scale AI across Clusters...
byAlluxio, Inc.
 
SWEBOK: the software engineering body of knowledge (SC25 Workshop Research So...
byHironori Washizaki
 
AOX Apps - Mobile App Development Company New York
byAOX APPS
 
NYC ACE 12-Nov-2025-Combined Presentation.pdf
byAUGNYC
 
DSD-INT 2025 Exploring different domain decomposition approaches for enhanced...
byDeltares
 
Communicating Software Architecture using Arc42 v1.1
byAshraf Fouad
 
Top Benefits of Hiring a Mobile App Development Company in India
byDevherds software solution
 
A Complete Guide to Salesforce for Education.pdf
byVALiNTRY360
 
How to Make Your AI Reliable: Comprehensive Guide to Testing AI Applications ...
byQATestLab Quality is a rule
 
From Chatbot to AI Agent: A Practical Comparison
byShiv Technolabs Pvt. Ltd.
 
SCORM Cloud: The 5 categories of content distribution
byRustici Software
 

Easy, scalable, fault tolerant stream processing with structured streaming - spark summit 2017

  • 1.
    Easy, Scalable, Fault-tolerant StreamProcessing with Structured Streaming Michael Armbrust - @michaelarmbrust Tathagata Das - @tathadas Spark Summit 2017 6th June, San Francisco
  • 2.
    TEAM About Databricks Started Sparkproject (now Apache Spark) at UC Berkeley in 2009 PRODUCT Unified Analytics Platform MISSION Making Big Data Simple
  • 3.
  • 4.
    Complexities in streamprocessing COMPLEX DATA Diverse data formats (json, avro, binary, …) Data can be dirty, late, out-of-order COMPLEX SYSTEMS Diverse storage systems (Kafka, S3, Kinesis, RDBMS, …) System failures COMPLEX WORKLOADS Combining streaming with interactive queries Machine learning
  • 5.
    Structured Streaming stream processingon Spark SQL engine fast, scalable, fault-tolerant rich, unified, high level APIs deal with complex data and complex workloads rich ecosystem of data sources integrate with many storage systems
  • 6.
    you should not haveto reason about streaming
  • 7.
    you should write simplequeries & Spark should continuously update the answer
  • 8.
    Streaming word count Anatomyof a Streaming Query
  • 9.
    Anatomy of aStreaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy($"value".cast("string")) .count() .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode(OutputMode.Complete()) .option("checkpointLocation", "…") .start() Source • Specify one or more locations to read data from • Built in support for Files/Kafka/Socket, pluggable. • Can include multiple sources of different types using union()
  • 10.
    Anatomy of aStreaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode(OutputMode.Complete()) .option("checkpointLocation", "…") .start() Transformation • Using DataFrames, Datasets and/or SQL. • Catalyst figures out how to execute the transformation incrementally. • Internal processing always exactly-once.
  • 11.
    DataFrames, Datasets, SQL input =spark.readStream .format("kafka") .option("subscribe", "topic") .load() result = input .select("device", "signal") .where("signal > 15") result.writeStream .format("parquet") .start("dest-path") Logical Plan Read from Kafka Project device, signal Filter signal > 15 Write to Kafka Spark automatically streamifies! Spark SQL converts batch-like query to a series of incremental execution plans operating on new batches of data Series of Incremental Execution Plans Kafka Source Optimized Operator codegen, off- heap, etc. Kafka Sink Optimized Physical Plan process newdata t = 1 t = 2 t = 3 process newdata process newdata
  • 12.
    Anatomy of aStreaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode(OutputMode.Complete()) .option("checkpointLocation", "…") .start() Sink • Accepts the output of each batch. • When supported sinks are transactional and exactly once (Files). • Use foreach to execute arbitrary code.
  • 13.
    Anatomy of aStreaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode("update") .option("checkpointLocation", "…") .start() Output mode – What's output • Complete – Output the whole answer every time • Update – Output changed rows • Append – Output new rows only Trigger – When to output • Specified as a time, eventually supports data size • No trigger means as fast as possible
  • 14.
    Anatomy of aStreaming Query spark.readStream .format("kafka") .option("subscribe", "input") .load() .groupBy('value.cast("string") as 'key) .agg(count("*") as 'value) .writeStream .format("kafka") .option("topic", "output") .trigger("1 minute") .outputMode("update") .option("checkpointLocation", "…") .start() Checkpoint • Tracks the progress of a query in persistent storage • Can be used to restart the query if there is a failure.
  • 15.
    Fault-tolerance with Checkpointing Checkpointing– tracks progress (offsets) of consuming data from the source and intermediate state. Offsets and metadata saved as JSON Can resume after changing your streaming transformations end-to-end exactly-once guarantees process newdata t = 1 t = 2 t = 3 process newdata process newdata write ahead log
  • 16.
  • 17.
    Traditional ETL Raw, dirty,un/semi-structured is data dumped as files Periodic jobs run every few hours to convert raw data to structured data ready for further analytics 17 file dump seconds hours table 10101010
  • 18.
    Traditional ETL Hours ofdelay before taking decisions on latest data Unacceptable when time is of essence [intrusion detection, anomaly detection, etc.] file dump seconds hours table 10101010
  • 19.
    Streaming ETL w/Structured Streaming Structured Streaming enables raw data to be available as structured data as soon as possible 19 seconds table 10101010
  • 20.
    Streaming ETL w/Structured Streaming Example Json data being received in Kafka Parse nested json and flatten it Store in structured Parquet table Get end-to-end failure guarantees val rawData = spark.readStream .format("kafka") .option("kafka.boostrap.servers",...) .option("subscribe", "topic") .load() val parsedData = rawData .selectExpr("cast (value as string) as json")) .select(from_json("json", schema).as("data")) .select("data.*") val query = parsedData.writeStream .option("checkpointLocation", "/checkpoint") .partitionBy("date") .format("parquet") .start("/parquetTable")
  • 21.
    Reading from Kafka Specifyoptions to configure How? kafka.boostrap.servers => broker1,broker2 What? subscribe => topic1,topic2,topic3 // fixed list of topics subscribePattern => topic* // dynamic list of topics assign => {"topicA":[0,1] } // specific partitions Where? startingOffsets => latest(default) / earliest / {"topicA":{"0":23,"1":345} } val rawData = spark.readStream .format("kafka") .option("kafka.boostrap.servers",...) .option("subscribe", "topic") .load()
  • 22.
    Reading from Kafka valrawData = spark.readStream .format("kafka") .option("kafka.boostrap.servers",...) .option("subscribe", "topic") .load() rawData dataframe has the following columns key value topic partition offset timestamp [binary] [binary] "topicA" 0 345 1486087873 [binary] [binary] "topicB" 3 2890 1486086721
  • 23.
    Transforming Data Cast binaryvalue to string Name it column json val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*")
  • 24.
    Transforming Data Cast binaryvalue to string Name it column json Parse json string and expand into nested columns, name it data val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*") json { "timestamp": 1486087873, "device": "devA", …} { "timestamp": 1486082418, "device": "devX", …} data (nested) timestamp device … 1486087873 devA … 1486086721 devX … from_json("json") as "data"
  • 25.
    Transforming Data Cast binaryvalue to string Name it column json Parse json string and expand into nested columns, name it data Flatten the nested columns val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*") data (nested) timestamp device … 1486087873 devA … 1486086721 devX … timestamp device … 1486087873 devA … 1486086721 devX … select("data.*") (not nested)
  • 26.
    Transforming Data Cast binaryvalue to string Name it column json Parse json string and expand into nested columns, name it data Flatten the nested columns val parsedData = rawData .selectExpr("cast (value as string) as json") .select(from_json("json", schema).as("data")) .select("data.*") powerful built-in APIs to perform complex data transformations from_json, to_json, explode, ... 100s of functions (see our blog post)
  • 27.
    Writing to Save parseddata as Parquet table in the given path Partition files by date so that future queries on time slices of data is fast e.g. query on last 48 hours of data val query = parsedData.writeStream .option("checkpointLocation", ...) .partitionBy("date") .format("parquet") .start("/parquetTable")
  • 28.
    Checkpointing Enable checkpointing by settingthe checkpoint location to save offset logs start actually starts a continuous running StreamingQuery in the Spark cluster val query = parsedData.writeStream .option("checkpointLocation", ...) .format("parquet") .partitionBy("date") .start("/parquetTable/")
  • 29.
    Streaming Query query isa handle to the continuously running StreamingQuery Used to monitor and manage the execution val query = parsedData.writeStream .option("checkpointLocation", ...) .format("parquet") .partitionBy("date") .start("/parquetTable")/") process newdata t = 1 t = 2 t = 3 process newdata process newdata StreamingQuery
  • 30.
    Data Consistency onAd-hoc Queries Data available for complex, ad-hoc analytics within seconds Parquet table is updated atomically, ensures prefix integrity Even if distributed, ad-hoc queries will see either all updates from streaming query or none, read more in our blog https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html complex, ad-hoc queries on latest data seconds!
  • 31.
    More Kafka Support[Spark 2.2] Write out to Kafka Dataframe must have binary fields named key and value Direct, interactive and batch queries on Kafka Makes Kafka even more powerful as a storage platform! result.writeStream .format("kafka") .option("topic", "output") .start() val df = spark .read // not readStream .format("kafka") .option("subscribe", "topic") .load() df.registerTempTable("topicData") spark.sql("select value from topicData")
  • 32.
    Amazon Kinesis [DatabricksRuntime 3.0] Configure with options (similar to Kafka) How? region => us-west-2 / us-east-1 / ... awsAccessKey (optional) => AKIA... awsSecretKey (optional) => ... What? streamName => name-of-the-stream Where? initialPosition => latest(default) / earliest / trim_horizon spark.readStream .format("kinesis") .option("streamName", "myStream") .option("region", "us-west-2") .option("awsAccessKey", ...) .option("awsSecretKey", ...) .load()
  • 33.
  • 34.
    Event Time Many usecases require aggregate statistics by event time E.g. what's the #errors in each system in the 1 hour windows? Many challenges Extracting event time from data, handling late, out-of-order data DStream APIs were insufficient for event-time stuff
  • 35.
    Event time Aggregations Windowingis just another type of grouping in Struct. Streaming number of records every hour Support UDAFs! parsedData .groupBy(window("timestamp","1 hour")) .count() parsedData .groupBy( "device", window("timestamp","10 mins")) .avg("signal") avg signal strength of each device every 10 mins
  • 36.
    Stateful Processing forAggregations Aggregates has to be saved as distributed state between triggers Each trigger reads previous state and writes updated state State stored in memory, backed by write ahead log in HDFS/S3 Fault-tolerant, exactly-once guarantee! process newdata t = 1 sink src t = 2 process newdata sink src t = 3 process newdata sink src state state write ahead log state updates are written to log for checkpointing state
  • 37.
    Automatically handles LateData 12:00 - 13:00 1 12:00 - 13:00 3 13:00 - 14:00 1 12:00 - 13:00 3 13:00 - 14:00 2 14:00 - 15:00 5 12:00 - 13:00 5 13:00 - 14:00 2 14:00 - 15:00 5 15:00 - 16:00 4 12:00 - 13:00 3 13:00 - 14:00 2 14:00 - 15:00 6 15:00 - 16:00 4 16:00 - 17:00 3 13:00 14:00 15:00 16:00 17:00Keeping state allows late data to update counts of old windows red = state updated with late data But size of the state increases indefinitely if old windows are not dropped
  • 38.
    Watermarking Watermark - movingthreshold of how late data is expected to be and when to drop old state Trails behind max seen event time Trailing gap is configurable event time max event time watermark data older than watermark not expected 12:30 PM 12:20 PM trailing gap of 10 mins
  • 39.
    Watermarking Data newer thanwatermark may be late, but allowed to aggregate Data older than watermark is "too late" and dropped Windows older than watermark automatically deleted to limit the amount of intermediate state max event time event time watermark late data allowed to aggregate data too late, dropped
  • 40.
    Watermarking max event time eventtime watermark allowed lateness of 10 mins parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() late data allowed to aggregate data too late, dropped Useful only in stateful operations (streaming aggs, dropDuplicates, mapGroupsWithState, ...) Ignored in non-stateful streaming queries and batch queries
  • 41.
    Watermarking data too late, ignoredin counts, state dropped Processing Time12:00 12:05 12:10 12:15 12:10 12:15 12:20 12:07 12:13 12:08 EventTime 12:15 12:18 12:04 watermark updated to 12:14 - 10m = 12:04 for next trigger, state < 12:04 deleted data is late, but considered in counts parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() system tracks max observed event time 12:08 wm = 12:04 10min 12:14 More details in my blog post
  • 42.
    Clean separation ofconcerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics Processing Details separated from
  • 43.
    Clean separation ofconcerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics How to group data by time? (same for batch & streaming) Processing Details
  • 44.
    Clean separation ofconcerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics How to group data by time? (same for batch & streaming) Processing Details How late can data be?
  • 45.
    Clean separation ofconcerns parsedData .withWatermark("timestamp", "10 minutes") .groupBy(window("timestamp","5 minutes")) .count() .writeStream .trigger("10 seconds") .start() Query Semantics How to group data by time? (same for batch & streaming) Processing Details How late can data be? How often to emit updates?
  • 46.
    Arbitrary Stateful Operations[Spark 2.2] mapGroupsWithState allows any user-defined stateful function to a user-defined state Direct support for per-key timeouts in event-time or processing-time Supports Scala and Java 46 ds.groupByKey(_.id) .mapGroupsWithState (timeoutConf) (mappingWithStateFunc) def mappingWithStateFunc( key: K, values: Iterator[V], state: GroupState[S]): U = { // update or remove state // set timeouts // return mapped value }
  • 47.
    Other interesting operations StreamingDeduplication Watermarks to limit state Stream-batch Joins Stream-stream Joins Can use mapGroupsWithState Direct support oming soon! val batchData = spark.read .format("parquet") .load("/additional-data") parsedData.join(batchData, "device") parsedData.dropDuplicates("eventId")
  • 48.
  • 49.
    Metric Processing @ DashboardsAnalyze trends in usage as they occur Alerts Notify engineers of critical issues Ad-hoc Analysis Diagnose issues when they occur ETL Clean, normalize and store historical data Events generated by user actions (logins, clicks, spark job updates)
  • 50.
    Metric Processing @ Dashboards Alerts Ad-hoc Analysis ETL Difficultwith only streaming frameworks Limited retention in streaming storage Inefficient for ad-hoc queries Hard for novice users (limited or no SQL support)
  • 51.
  • 52.
    Read from rawLogs =spark.readStream .format("kafka") .option("kafka.bootstrap.servers", ...) .option("subscribe", "rawLogs") .load() augmentedLogs = rawLogs .withColumn("msg", from_json($"value".cast("string"), schema)) .select("timestamp", "msg.*") .join(table("customers"), ["customer_id"]) DataFrames can be reused for multiple streams Can build libraries of useful DataFrames and share code between applications JSON ETL
  • 53.
    Write to augmented .repartition(1) .writeStream .format("parquet") .option("path", "/data/metrics") .trigger("1minute") .start() Store augmented stream as efficient columnar data for later processing Latency: ~1 minute Buffer data and write one large file every minute for efficient reads ETL
  • 54.
    Dashboards logins = spark.readStream.parquet("/data/metrics") .where("metric= 'login'") .groupBy(window("timestamp", "1 minute")) .count() display(logins) // Visualize in Databricks notebooks Always up-to-date visualizations of important business trends Latency: ~1 minute to hours (configurable) Dashboards
  • 55.
    Filter and writeto filteredLogs = augmentedLogs .where("eventType = 'clusterHeartbeat'") .selectExpr("to_json(struct("*")) as value") filteredLogs.writeStream .format("kafka") .option("kafka.bootstrap.servers", ...) .option("topic", "clusterHeartbeats") .start() Forward filtered and augmented events back to Kafka Latency: ~100ms average Filter to_json() to convert columns back into json string, and then save as different Kafka topic
  • 56.
    Simple Alerts sparkErrors .as[ClusterHeartBeat] .filter(_.load >99) .writeStream .foreach(new PagerdutySink(credentials)) E.g. Alert when Spark cluster load > threshold Latency: ~100 ms Alerts Notify PagerDuty
  • 57.
    Complex Alerts sparkErrors .as[ClusterHeartBeat] .groupBy(_.id) .flatMapGroupsWithState(Update, ProcessingTimeTimeout("1minute")) { (id: Int, events: Iterator[ClusterHeartBeat], state: GroupState[ClusterState]) => ... // check if cluster non-responsive for a while } E.g. Monitor health of Spark clusters using custom stateful logic Latency: ~10 seconds Alerts React if no heartbeat from cluster for 1 min
  • 58.
    Ad-hoc Analysis SELECT * FROMparquet.`/data/metrics` WHERE level IN ('WARN', 'ERROR') AND customer = "…" AND timestamp < now() – INTERVAL 1 HOUR Trouble shoot problems as they occur with latest information Latency: ~1 minute Ad-hoc Analysis will read latest data when query executed
  • 59.
    Metric Processing @ = Metrics Filter ETL Dashboards Ad-hoc Analysis Alerts14+billion records / hour with 10 nodes meet diverse latency requirements as efficiently as possible
  • 60.
    More Info Structured StreamingProgramming Guide http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html Databricks blog posts for more focused discussions https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html https://databricks.com/blog/2017/01/19/real-time-streaming-etl-structured-streaming-apache-spark-2-1.html https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html https://databricks.com/blog/2017/04/26/processing-data-in-apache-kafka-with-structured-streaming-in-apache-spark-2-2.html https://databricks.com/blog/2017/05/08/event-time-aggregation-watermarking-apache-sparks-structured-streaming.html and more to come, stay tuned!!
  • 61.
    UNIFIED ANALYTICS PLATFORM TryApache Spark in Databricks! • Collaborative cloud environment • Free version (community edition) DATABRICKS RUNTIME 3.0 • Apache Spark - optimized for the cloud • Caching and optimization layer - DBIO • Enterprise security - DBES Try for free today databricks.com