Reactconf 2014 - Event Stream Processing

{MAKING SENSE OF THE FIRE-HOSE IN REAL-TIME}
{EVENT PROCESSING}
Dr Andy Piper
Push Technology
Reappt, a Push Technology product offers the enterprise grade Diffusion technology as a
service.

Time @cobbscomedyclub
The past, the present and the future walked
Copyright Push Technology 2014
into a bar
It was tense

Will the real Andy Piper please stand up?
– CTO at Push Technology
– Ex-BEA/Oracle
– Spring contributor (Spring DM) and Author
– Standards contributor – OMG, JCP etc
– PhD, Cambridge, Distributed Systems
✗ ✗

Agenda in 140 characters
What is it - What not? Why? History. Measure
infinity. Windows. Queries. Going fast –
reliably, distributed, distributed and fast and big

What is Event Stream Processing?

• It’s not stream processing
– Typically focused on local parallelism
– I have opinions but they get me in trouble

• Not event passing
– Event exchange not processing, e.g. JMS
– Stateless

• Not event mediation (brokering)
– Filtering, routing, and enrichment, e.g. ESB
– Stateless

“Event Stream Processing deals with the
task of processing streams of event data
with the goal of identifying the meaningful
pattern within those streams” – Wikipedia

• ESP is about querying data
streams
– Looking for something
– Haystack won’t stay still!
– Answers depend on multiple events
– Extremely stateful
Where the interesting questions
are!

Meta-analogy
“Producing thrust with a scramjet has been
“Event stream processing is like looking for
compared to lighting a match in a hurricane
a needle in a haystack in a hurricane”
and keeping it burning” - NASA

It’s like an inverted database
• Data is ‘static’
• Queries are ‘dynamic’
Query Event
• Data is ‘dynamic’
• Queries are ‘static’
RDBMS CEP
Data Query

Why bother?
• Too much data
• Time is integral to the questions
• Data is moving too fast
• Databases assume static datasets
?

History – Two schools of thought
• Database and make it time driven
• Logic approach with time constraints

Stream Processing History
• Tapestry – ’92
– Early inverted database (not Apache!)
• Materialized views – ‘95
– [A. Gupta and I. S. Mumick. “Maintenance of materialized views: Problems,
techniques, and applications.” 1995]
• David Luckham coined term CEP – “The Power Events”. 2001
– Logic-based CEP
– Company acquired by Avaya
• Michael Franklin
– Dataflow processing in PostgreSQL
– [“TelegraphCQ: continuous dataflow processing.” 2003]
• Aurora – ‘03
– [Cherniack et al – “Scalable distributed stream processing.” 2003]
• STREAM – ‘03
– [Arasu et al – “STREAM: The Stanford Stream Data Manager.” 2003]
• Borealis – ‘05
– [Abadi et al – “The design of the Borealis stream processing engine.” 2005]

Some definitions
• Tuple – a multi-set of elements ( e1, e2, … en )
– A single tuple is a monad!
• Event or Data Stream 푺풏 - any ordered pair 풔, Δ 풏
– 푠 is an unbounded sequence of tuples and
– Δ is an unbounded sequence of positive real time
intervals
– 푠 and Δ are of equal length
• Event stream processing transforms event streams
into new event streams through queries
• Outputs and inputs continuous
– Operators are continuous queries

How do you measure infinity?
How do you measure an event
stream if it’s unbounded?

Measuring infinity
• Don’t do it
– But just event passing – where is the fun in
that?!
• Synopses – store summary information
– Continuous average = running total + items
• Windows – define working set
– Continuous average over last N items

Measuring infinity

Types of window
• Sliding
• Jumping (batching)
• Partitioned
• Time-based
• Others

What to do with a working set?
• Windows define the scope of interest
• Run queries against working set as it
changes
– Continuous Queries

When should you run queries?
• Run queries when output is not
idempotent
• When is that?
– Contents of the window changes – maybe?
– Time advances – possibly?
– Depends on window and query
Linking cause and effect in an efficient manner
lies at the heart of CEP and is why the answer
is not simply programming

How can we define queries on windows?
• Describe queries on windows using a
SQL-like syntax
SELECT AVG(price) FROM stream [ROWS N]
• [Arasu et al. – “The CQL Continuous Query
Language: Semantic Foundations and Query
Execution” 2003]

Querying windows
• Sliding
SELECT * FROM s [ROWS 4 SLIDE 4]
• Partitioned
SELECT a, b FROM s
[PARTITION BY b ROWS 3]
• Time-based
SELECT * FROM s [RANGE 30 SECONDS]

How do you make it fast?
• Generally in-memory the only way
• Operate as a gigantic state machine and
optimize like crazy
– Go reactive!
– Talk to Martin

Why must it be fast?
• Not reactive streams!
• Flow control causes causal paradox
• Stream processing must keep up

How do you make it resilient?
• Making stateful systems resilient has
challenges
• State generally changing extremely quickly

Resiliency approaches
• Save all the things and replay
– But infinite data?!
– Sometimes possible because
append-only
• Save all the state
– Assumes there is less of it
– State is changing rapidly
– Too rapid to be effective

Resiliency approaches
• Elsewhere checkpoint and record changes
– Maybe we can record state and things
– Many commercial systems do
• No recording - identical parallel systems
– Synchronization an issue
– Catch-up an issue

How do you scale stream processing?
• Follow the crowd
• Distribute processing
• Multiple input sources
– If independent
– Flume
– Kafka

How do you distribute stream processing?
• DAG of event streams
– Inputs and outputs are event streams
– Nodes are operators or groups of operators
– Nodes can be distributed

Apache Storm
• Toolkit for creating distributed event flows
• Bolts (operators) and spouts (sources)
• Composed using a Clojure DSL
• Storm runs topologies
– Map-Reduce jobs finish – batch
– Topologies process forever – continuous

Apache Storm – a toolkit for distribution
(topology
{"1" (spout-spec twitter-feed-spout)}
{"2" (bolt-spec {"1"} filter :p "status" )}
{"3" (spout-spec database :p "retail" )}
{"4" (bolt-spec {"2"} top-n)}
{"5" (bolt-spec {"3" "4"} join :p "item" )}
...
)

How do you reliably distribute?
• State is now distributed
– Synchronization all but impossible
– Deterministic if relative order is preserved
• Depends on operators and their effect
• [L. Lamport - “Time, Clocks, and the Ordering of
Events in a Distributed System.” 1978]
– In theory a replay of things through the
network will recover the state
– Alternative of storing the state for all the
operators is harder

How do you reliably distribute?
• Different classes of recovery
– [Hwang et al. – “High-Availability Algorithms
for Distributed Stream Processing”. 2005]
• Precise recovery – failure effects hidden perfectly
• Rollback recovery – no data loss, but outputs
may be duplicated
• Gap recovery – data lost during recovery
• Reliable distribution overlaps distribution
– Upstream backup, reactive streams?

Reactive stream processing
• Message/event driven
• Discussed resiliency
• Continuous queries == responsive
– Push towards on-line queries
• Elasticity – harder

Stream Processing with Data
• Time dimension to data problems
• Data dimension to stream problems
• JOIN streams to tables
• Easy when small
• Large datasets harder
– Cache join data in memory?
– Push query into datastore?

Stream Processing with Big Data
• Time dimension to Big Data problems
– Velocity (vvv) implies stream processing
• Large dataset problem domain
• But now the data is distributed!

Shortcomings of Big Data

Fast Data Architecture

Fast Data Architecture
• Similar to recoverable architectures:
– Snapshot (queries) + incremental updates
– Current state = known state + changes
– Requires static queries - cached results
• Spark does this quite well

Fast data technology
• Storm – topology deployment
• Spark – logic queries on RDDs
• Spark streaming
– repeating snapshots / micro-batch
– Fast data-ish
• Flume – fast ingest of log data
• Kafka
– pub-sub messaging as distributed commit log
• Hadoop streaming
– create M-R jobs using executable scripts
• Hive
• Cloudera Impala
– MPP SQL query engine on top of Hadoop

Summary

Future Stream Processing
• Ease-of-use
– CQL or Graphical - both have drawbacks
– Queries get really complicated really quickly
• Ease-of-use + distribution
– Real systems challenge
• Fast data architectures
• Real-time machine learning
– Spark ML Library
– Hadoop Mahout
• Interactive streaming queries – declarative and
caching
– Hive and Spark

Reactconf 2014 - Event Stream Processing

More Related Content

What's hot

Viewers also liked

Similar to Reactconf 2014 - Event Stream Processing

Recently uploaded

Reactconf 2014 - Event Stream Processing

Editor's Notes