Uploaded byDataWorks Summit
Bridging Batch and Real-time Systems for Anomaly Detection
This document discusses using a stack of Hadoop, Spark, and Elasticsearch to perform anomaly detection on large datasets in both batch and real-time. Hadoop is used for large-scale data storage and preprocessing. Spark is used to perform in-depth analysis to identify common entities and build models. Elasticsearch allows searching the data in real-time and performing aggregations to identify uncommon entities. A live loop continuously adapts the models to react to streaming data and improve anomaly detection over time.
More Related Content
Similar to Bridging Batch and Real-time Systems for Anomaly Detection
![[Vancouver] part 2 understanding the relevance of your search with elasticse...](https://cdn.slidesharecdn.com/ss_thumbnails/vancouverpart2understandingtherelevanceofyoursearchwithelasticsearchandkibana-210218165954-thumbnail.jpg?width=640&height=640&fit=bounds)
More from DataWorks Summit
![Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]](https://cdn.slidesharecdn.com/ss_thumbnails/agenticcommunityseries-day3-cfd-251120170304-ddef8112-thumbnail.jpg?width=640&height=640&fit=bounds)
Bridging Batch and Real-time Systems for Anomaly Detection
- 1. Bridging Batch andReal-time Systems for Anomaly Detection Costin Leau @costinl
- 3. www.elastic.co Interesting != Common Datasetstend to have hot / common entities Monopolize the data set Create too much noise Cannot be easily avoided Common = frequent Interesting = frequently different
- 4. www.elastic.co Finding the uncommon Backgroundvs foreground == things that stand out Example: Background: “flu” “H5N1” appears in 5 / 10M docs H5N1 flu
- 5. www.elastic.co Finding the uncommon Backgroundvs foreground == things that stand out Example: Background: “flu” “H5N1” appears in 5 / 10M docs Foreground: “bird flu” “H5N1” appears in 4 / 100 docs H5N1 bird flu H5N1 flu
- 6.
- 7. www.elastic.co Great for bulkdata Does not require prepared data Useful for building data models Execution options Long-running (Batch?) Time-sensitive operations Rich answers rely on metadata Work great with data streams Real-time
- 8. www.elastic.co Finding the uncommon- Stack Deal with big data sets Hadoop / Spark Perform the analysis & exploration Elasticsearch Mine the data Spark Elasticsearch
- 9. The Big Picture HDFS Slow,in-depth (machine) learning Fast, real-time learning ETL
- 10. www.elastic.co Hadoop De-facto platform forbig data HDFS - Used for storing and performing ETL at scale YARN – Job scheduling and resource management
- 11. www.elastic.co Apache Spark Apache Spark™is a fast and general engine for large-scale data processing Provides building blocks for fast reads and transformations streaming (Spark Streaming) machine learning (MLLib) works great with Hadoop (HDFS and YARN)
- 12. www.elastic.co Elasticsearch Open-source real-time searchand analytics engine • Fully-featured search Relevance-ranked text search Scalable search High-performance geo, temporal, range and key lookup Highlighting Support for complex / nested document types * Spelling suggestions Powerful query DSL * “Standing” queries * Real-time results * Extensible via plugins * • Powerful faceting/analysis Summarize large sets by any combinations of time, geo, category and more. * “Kibana” visualization tool * * Features we see as differentiators • Management Simple and robust deployments * REST APIs for handling all aspects of administration/monitoring * “Marvel” console for monitoring and administering clusters * Special features to manage the life cycle of content * • Integration Hadoop (Map/Red,Hive,Pig,Cascading..)* Client libraries (Python, Java, Ruby, javascript…) Data connectors (Twitter, JMS…) Logstash ETL framework * • Support Development and Production support with tiered levels Support staff are the core developers of the product *
- 14.
- 15.
- 16.
- 17.
- 18.
- 19.
- 20.
- 21.
- 22.
- 23.
- 24. www.elastic.co Inverted index Inverting Shakespeare Takeall the plays and break them down word by word For each word, store the ids of the documents that contain it Sort all tokens (words) token doc freq. postings (doc ids) Anthony 2 1, 2 Brutus 1 5 Caesar 2 2, 3 Calpurnia 2 4, 5
- 25. www.elastic.co Relevancy How well doesa document match a query? step query d1 d2 The text brown fox The quick brown fox likes brown nuts The red fox The terms (brown, fox) (brown, brown, fox, likes, nuts, quick) (red, fox) A frequency vector (1, 1) (2, 1) (0, 1) Relevancy - 2? 1?
- 26. www.elastic.co Relevancy - VectorSpace Model How well q matches d1 and d2? The coordinates in the vector represent weights per term The simple (1, 0) vector defines these weights based on the frequency of each term But to generalize: . 2 1 1 tf: brown tf: fox q: (brown, fox) d1: (brown, brown, fox) d2: (fox)
- 27. www.elastic.co Relevancy – TF/IDF TermFrequency / Inverse Document Frequency TF = the more a token appears in a doc, the more important it is IDF = the more documents containing the term, the less important it is
- 28. www.elastic.co Called Lucene Similarity Canbe ignored (was an attempt to make query scores comparable across indices, it’s there for backward compatibility) Core TF/IDF weight Score of a document for a given query Normalized doc length, shorter docs are more likely to be relevant than longer docs Boost of query term t Ranking Formula
- 29.
- 30. www.elastic.co Frequency differentiator TF-IDF by-itselfis not enough need to compare the DF in foreground vs background Precision vs Recall balance
- 31. www.elastic.co Single-set analysis A CF H I K A B C D E … X Y Z W Query results Dataset
- 32. www.elastic.co Single-set analysis example crimes bicycle theft crimes bicycle theft BritishPolice Force British Transport Police
- 33. www.elastic.co Multi-set analysis A BC D E … X Y Z W A C F H I K M Q R … Query results Dataset A B C D .. J L M N O .. U Aggregate
- 34.
- 35.
- 36. www.elastic.co Hadoop / Spark Off-line/ “slow” learning In-depth analysis Break down data into hot spots Eliminate noise Build multiple models
- 37. www.elastic.co Elasticsearch Search features Scoring, TF-IDF Significantterms (multi-set analysis) Aggregations Buckets & Metrics
- 38.
- 39. www.elastic.co Reacting to livedata Preventing execute queries as the data flows in Routing place suspicious data into a dedicate pipeline
- 40.
- 41. www.elastic.co Live loop Data keepschanging Adapt the set of rules Improves reaction time Build a model for fast decision making Keeps the prevention rate high Categorize data on the fly Elasticsearch Streaming
- 42. www.elastic.co Finding interesting data– basic approach
- 43. www.elastic.co Finding interesting data- analytics
- 44. www.elastic.co Finding interesting data- through a ML model
- 45. www.elastic.co Q & A Thankyou! @costinl
Editor's Notes
- #8 Conceptually they are the same, same data structure, same APIs but the usage model differs.
- #14 Bullet 1: As user content is created, its indexed and available for others to see in their search results in realtime automatically - its even relevant to them based on boosting, scoring etc. Bullet 2: New features can go out due to the Schemaless or Schema-lite nature of Elasticsearch. No need to change the schema, just update and go as you develop the document and application.
- #19 Including highlighting, DYM
- #20 filtering between dates/ranges
- #31 Precision = how many of the retrieved documents are relevant Recall = how many of the relevant documents are retrieved
- #35 Term aggregation
- #36 Significant terms