Real-Time Analytics with Kafka, Cassandra and Storm

Real-Time Analytics with Kafka, Cassandra and Storm

• 1.
  REAL-TIME ANALYTICS WITH KAFKA,CASSANDRA & STORM Dr. John Georgiadis Modio Computing
• 2.
  Modio Computing USE CASES •Collecting/processing measurements from large sensor networks (e.g. weather data). • Aggregated processing of financial trading streams. • Customer activity monitoring for advertising purposes, fraud detection, etc. • Real-time security log processing.
• 3.
  Modio Computing SOLUTION APPROACH •Real-time Updates: Employ streaming instead of batch analytics. • Apache Storm: Large installation base. Streaming & micro-batch. • Apache Spark: Uniform API for batch & micro-batch. On top of YARN/HDFS. Micro-batch less mature but catching-up quickly. • Large data sets +Time Series + Write-Intensive + Data Expiration = Apache Cassandra
• 4.
  Modio Computing ARCHITECTURE
• 5.
  Modio Computing APACHE KAFKA •N Nodes • TTopics • Replication Factor: Defines high availability • Partitions:They define parallelism level.A single consumer per partition. • Consumer discovers cluster nodes through Zookeeper • Consumer partition state is just an integer: the partition offset.
• 6.
  Modio Computing STORM • Stormis a distributed computing platform. In Storm a distributed computation is a directed graph of interconnected processors (topology) that exchange messages. • Spouts: Processors that inject messages into the topology. • Bolts: Processors that process messages including sending to 3rd parties (.e.g persistence). • Trident: High-level operations on message batches. Support batch replay in case of failure.Translates to a graph of low-level spouts and bolts.
• 7.
  Modio Computing STORM ::NIMBUS • A single controller (Nimbus) 
 where topologies are submitted. 
 Nimbus breaks topologies in tasks and 
 forwards to supervisors which spawn 
 one or more workers(processes) per 
 task. Nimbus redistributes tasks in case a supervisor fails. Nimbus is not HA. If Nimbus fails, running topologies are not affected.
• 8.
  Modio Computing STORM ::SUPERVISOR • 1 supervisor per host. • Supervisor registers with ZK at startup and thus it’s discoverable by Nimbus. • Supervisor spawns Worker JVMs: one process per topology. • JAR submitted to Nimbus is copied to Worker classpath. • When a Supervisor dies, all Worker tasks are migrated to the remaining Supervisors.
• 9.
  Modio Computing STORM ::TRIDENT •When to use Micro-batch (akaTrident) instead 
 of Streaming. • Millisecond latency not required.Typical 
 Trident latency threshold: 500ms. Allows batch mode persistence operations. High-level abstractions: partitionBy, partitionAggregate, stateQuery, partitionPersist. Batch processing timeout/exception will cause a replay of the batch provided the Spout supports replays (Kafka does):At-least-once semantics.
• 10.
  Modio Computing STORM ::PARALLELISM • Parallelism = Number of threads executing a topology cluster-wide. • Parallelism <= CPU threads/worker x Workers • Define per-topology max #workers (explicitly) and max parallelism (implicitly). • Define explicitly topology step parallelism. 
 Max parallelism = Σ(step parallelism). • Trident merges multiple steps into the same thread/node. Last parallelism statement is the effective parallelism. • Repartition operations define step merging boundaries. • Repartition operations (shuffle, partitionBy, broadcast, etc.) imply network transfer and are expensive. In some cases they are disastrous to performance!
• 11.
  Modio Computing STORM ::PERFORMANCETUNING • Spouts must match upstream parallelism: one spout per Kafka partition. • Little’s Law: Batch Size =Throughput x Latency • Adjust batch size: (Kafka Partitions) x (Kafka fetch size)
 Larger batch size = {higher throughput, higher latency}
 Increase batch size gradually until latency starts increasing sharply. • Identify the slowest stage (I/O or CPU bound): • You can’t have better throughput than the throughput of your slowest stage. • You can’t have better latency than the sum of individual latencies. • If CPU bound, increase parallelism. If I/O bound increase downstream (i.e storage) capacity.
• 12.
  Modio Computing CASSANDRA ::THEGOOD • Great write performance. Decent read performance. • Write latency: 20μs-120μs • 15K writes/sec on a single node • Extremely stable.Very low record of data corruption. • Decentralized setup: all cluster nodes have the same setup. • Multi-datacenter setups. • Configurable consistency of updates: ONE, QUORUM,ALL. • TTL per cell (row & column). • Detailed metrics: #operations, latencies, thread pools, memory, cache performance.
• 13.
  Modio Computing CASSANDRA ::THE“FEATURES” • All partition keys must be set in queries. • All primary keys preceding an initialized primary key with a value must also be initialized in queries. • TTL is not allowed on cells containing counters. • NULL values are not supported on primary keys. • Range queries can only be applied on the last column of the composite primary key that appears in the query. • Disjunction operator (OR) is not available.The IN keyword can be used in some cases instead. • Row counting is a very expensive operation.
• 14.
  Modio Computing CASSANDRA ::PERFORMANCE • Design the schema around the partition key. • Keep each partition size small (no more than a few 100s entry) as reading will fetch the whole partition. • Leverage Key cache • Avoid making time fragments part of the partition key as this will direct all activity to the node that is the partition owner at a given date. • Query/Update Plan: • Avoid range queries and the IN operator as it requires contacting multiple nodes and assembling the results at the coordinator node. • Use prepared statements to avoid repeated statement parsing. • Prefer async writes combined with a max pending statements threshold. • Best performance out of batches containing statements with the same partition key.
• 15.
  Modio Computing CASSANDRA ::CLUSTER • One or more data nodes are also “seed”nodes 
 acting as the membership gatekeepers. • Table sharding across the cluster based on the 
 partition key hash (token). • Table replication according to replication 
 factor (RF). Configurable per keyspace (database). • The Java driver has several load balancing approaches: • Token-aware: sends each statement to the node that actually will store it. Random selection amongst the nodes for a given replica set. • Latency-aware: sends each statement to the node with the fastest response. • Round-robin & custom load balancers supported.
• 16.
  Modio Computing • Kafka •N-way replication: N-1 node failures. • Clients dynamically reconfigured if accessing through Zookeeper. • Storm • For cluster size N, X supervisor failures provided (N-X) nodes have memory 
 to accommodate X JVM worker processes. • Incomplete batches replayed:At-least-once semantics. • Cassandra • N-way replication: N-1 node failures if using ONE consistency, if using QUORUM consistency. • Clients require a list of all cluster nodes. • Zookeeper • Majority voting is required:At most F failures in cluster with 2F+1 nodes. Leader re-election very fast (200ms). • Sizes bigger than 3-5 not recommended due to decreasing write performance. • If majority voting is lost, Storm will stop. Kafka will fail to commit client offsets. If majority is regained Storm will resume. Kafka brokers will resume in most cases. FAILURE SCENARIOS