Turi, Inc., profile picture
Uploaded byTuri, Inc.
841 views

Pattern Mining: Extracting Value from Log Data

Pattern mining is an unsupervised machine learning technique used to discover frequent patterns and relationships in log data. It involves finding the top frequent sets of items that occur together in the data at least a minimum number of times. There are two main approaches - candidate generation which generates and filters candidate patterns in multiple passes over the data, and pattern growth which constructs conditional databases to avoid multiple full scans. Pattern mining can be used to find commonly purchased itemsets, extract features from log data, and derive rules for recommendations.

Embed presentation

Downloaded 32 times
Pattern Mining: Getting the most out of your log data. Krishna Sridhar Staff Data Scientist, Dato Inc. krishna_srd
• Background - Machine Learning (ML) Research. - Ph.D Numerical Optimization @Wisconsin • Now - Build ML tools for data-scientists & developers @Dato. - Help deploy ML algorithms. @krishna_srd, @DatoInc About Me!
45+$and$growing$fast! About Us!
+ = Questions? • (Now) We are monitoring the chat window. • (Later) Email me srikris@dato.com. Webinars
About you?
Creating a model pipeline Ingest Transform Model Deploy Unstructured Data exploration data modeling Data Science Workflow Ingest Transform Model Deploy
GraphLab(Create( Train Model Pipeline Deploy Models Serve Requests (REST API) Monitor Services Get Live Feedback Update Pipelines Prototype & Develop Model Pipelines Update Live Experiment Deploy New Pipeline Dato(Predic2ve(Services( Dato’s Products Dato(Distributed( We can help!
Log Journey Lots of data Insights Profits
Log Mining: Pattern Mining
Logs are everywhere!
Machine Learning in Logs Source: Mining Your Logs - Gaining Insight Through Visualization
Coffee shop Coffee Shops Menu
Receipts Coffee Shops Menu
Coffee Store Logs
Frequent Pattern Mining What sets of items were bought together?
Real Applications
Real Applications
Real Applications
Log Mining: Rule Mining
Can we recommend items? Rule Mining
Real Applications
Log Mining: Feature Extraction
Feature Extraction 0 1 0 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 0 1 1 1 0 Receipt Space Features in Menu Space ML
3 Useful Data Mining Tasks Rule MiningPattern Mining Feature Extraction
Demo
ML is not a black-box. Transparency Learning is also about understanding. Interpretability Whatever can go wrong, will go wrong. Diagnosis Moving on
Pattern Mining Explained
Formulating Pattern Mining N distinct items → 2N itemsets
Formulating Pattern Mining Find the top K most frequent sets of length at least L that occur at least M times.
Formulating Pattern Mining Find the top K most frequent sets of length at least L that occur at least M times. - max_patterns - min_length - min_support
Pattern Mining N distinct items → 2N itemsets
Pattern Mining: Principles
Principle 1: What is frequent? A pattern is frequent if it occurs at least M times. {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {C, D}: 5 is frequent M = 4 {A, D}: 5 is not frequent
Principle 1: What is frequent? A pattern is frequent if it occurs at least M times. {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {C, D}: 5 is frequent M = 4 {A, D}: 5 is not frequent min_support
Principle 2: Apriori principle A pattern is frequent only if a subset is frequent {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {B, C, D} : 5 is frequent therefore {C, D} : 5 is frequent {A} : 3 is not frequent therefore {A, D} : 3 is not frequent M = 4
Two Main Algorithms • Candidate Generation - Apriori - Eclat • Pattern Growth - FP-Growth - TopK FP-Growth [GLC 1.6]
Lots of Generalizations Source: http://www.philippe-fournier-viger.com/spmf/
Candidate Generation Two phases 1. Candidate generation. 2. Candidate filtering. Exploit Apriori Principle!
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : 4 {BD} : 4 {CD} : 5 {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Candidate Generation {AB} : ? {AC} : ? {AD} : ? {BC} : 4 {BD} : 4 {CD} : 5 {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
Pattern Growth Two phases 1. Candidate filtering 2. Conditional database constructions. Avoid full scans over the data & large candidate sets!
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : 1 {AC} : 2 {AD} : 3 {BD} : 4 {CD} : 4 {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : 0 {ABD} : 1 {ACD} : 2 {BCD} : 2 {BC} : 2
Pattern Growth - Preprocessing {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
Pattern Growth - Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : 2
Pattern Growth {B} : 4 { } : 6 Call: Growth(db = DB{}, item = B, freq = {B,C,D}) DB{} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D}
Pattern Growth {B} : 4 { } : 6 Conditional Database Construction DB{} DB{B} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {C, D} {D} {C, D} {D}
Pattern Growth {B} : 4 { } : 6 Candidate Filtering DB{B} {C, D} {D} {C, D} {D} {D} : 4 {C} : 2 DB{} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} DB{B} Add {BD} as frequent
Pattern Growth - Depth First {C, D} {D} {C, D} {D} {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : 2
Pattern Growth Recurse: Growth(db = DB{B}, item = D, freq = {D}) DB{B} {C, D} {D} {C, D} {D} {B} : 4 { } : 6 {BD} : 4 DB{BD}
Pattern Growth - Depth First {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : X {ACD} : ? {BCD} : X {BC} : 2
Compare & Constrast • Candidate Generation + Better than brute force + Filters candidate sets - Multiple passes over the data • Pattern Growth + Fewer passes over the data + Space efficient.
Compare & Constrast • Candidate Generation + Better than brute force + Filters candidate sets - Multiple passes over the data • Pattern Growth + Fewer passes over the data + Space efficient. Better choice
FP-Tree Compression Figures From Florian Verhein’s Slides on FP-Growth
FP-Growth Algorithm Figures From Florian Verhein’s Slides on FP-Growth Two phases 1. Candidate filtering. 2. Conditional database constructions.
TopK FP-Growth Algorithm Similar to FP-Growth 1. Dynamically raise min_support. 2. Estimates of min_support greatly help.
Performance on Website Logs • 1.5m events • 84k sessions • 3k unique ids
Future Work
Distributed FP-Growth Partition database on item-ids. Database
Bags + Sequences × 2 Itemset: {Item} Bags: {Item: quantity} Sequences : (item)
Model built, now what?
Creating a model pipeline Ingest Transform Model Deploy Unstructured Data exploration data modeling Data Science Workflow Ingest Transform Model Deploy
Demo
Summary Log Data Mining ≠ Rocket Science • FP-Growth for finding frequent patterns. • Find rules from patterns to make predictions. • Extract features for useful ML in pattern space.
SELECT questions FROM audience WHERE difficulty == “Easy” Thanks!
Extra Slides

More Related Content

PPT
Slides
bybutest
 
PPTX
Webinar - Pattern Mining Log Data - Vega (20160426)
byTuri, Inc.
 
PPTX
Webinar - Analyzing Video
byTuri, Inc.
 
PPTX
Intelligent Applications with Machine Learning Toolkits
byTuri, Inc.
 
PDF
Cloudera Impala technical deep dive
byhuguk
 
PPT
Swimming
byguestcdd2928
 
PPTX
A2 Sociology: Defining Relgion
byApril Lennox-Hill's Sociology Lessons
 
PPT
Steps involved in Sugar Production
bymustafeex
 
Slides
bybutest
 
Webinar - Pattern Mining Log Data - Vega (20160426)
byTuri, Inc.
 
Webinar - Analyzing Video
byTuri, Inc.
 
Intelligent Applications with Machine Learning Toolkits
byTuri, Inc.
 
Cloudera Impala technical deep dive
byhuguk
 
Swimming
byguestcdd2928
 
A2 Sociology: Defining Relgion
byApril Lennox-Hill's Sociology Lessons
 
Steps involved in Sugar Production
bymustafeex
 

Viewers also liked

PPT
Types of brand
byChao Onlamai
 
PPTX
Product life cycle & marketing strategy
byHitesh Sunny
 
PDF
10 Ways to Improve Internal Communication
byWeekdone.com
 
PDF
Congratulations Graduate! Eleven Reasons Why I Will Never Hire You.
byMark O'Toole
 
PPTX
Product life cycle & marketing strategies
byAmar Ingale
 
PPTX
XII Marketing Project Work
byRahil Jain
 
PDF
The Little Book of IDEO: Values
byTim Brown
 
PDF
Death by PowerPoint
byAlexei Kapterev
 
PPTX
Culture
byReed Hastings
 
PPTX
Slideshare ppt
byMandy Suzanne
 
PPTX
Eagle from eBay at China Hadoop Summit 2015
byHao Chen
 
PPTX
Developing Distributed Web Applications, Where does REST fit in?
bySrinath Perera
 
PPTX
Hadoop Eagle - Real Time Monitoring Framework for eBay Hadoop
byDataWorks Summit
 
PPTX
Siddhi: A Second Look at Complex Event Processing Implementations
bySrinath Perera
 
PDF
From Beginners to Experts, Data Wrangling for All
byDataWorks Summit
 
PDF
Apache Eagle - Monitor Hadoop in Real Time
byDataWorks Summit/Hadoop Summit
 
PPTX
Ahead Week 1 Key Slides
byaltonbaird
 
ODP
Animales en peligro de extincion
bylosdonkey
 
PPTX
BNI Lake Business Builders- LOZ Vice President report
byMike Tobin
 
PPTX
Setting up Your LinkedIn Account
byNET:101
 
Types of brand
byChao Onlamai
 
Product life cycle & marketing strategy
byHitesh Sunny
 
10 Ways to Improve Internal Communication
byWeekdone.com
 
Congratulations Graduate! Eleven Reasons Why I Will Never Hire You.
byMark O'Toole
 
Product life cycle & marketing strategies
byAmar Ingale
 
XII Marketing Project Work
byRahil Jain
 
The Little Book of IDEO: Values
byTim Brown
 
Death by PowerPoint
byAlexei Kapterev
 
Culture
byReed Hastings
 
Slideshare ppt
byMandy Suzanne
 
Eagle from eBay at China Hadoop Summit 2015
byHao Chen
 
Developing Distributed Web Applications, Where does REST fit in?
bySrinath Perera
 
Hadoop Eagle - Real Time Monitoring Framework for eBay Hadoop
byDataWorks Summit
 
Siddhi: A Second Look at Complex Event Processing Implementations
bySrinath Perera
 
From Beginners to Experts, Data Wrangling for All
byDataWorks Summit
 
Apache Eagle - Monitor Hadoop in Real Time
byDataWorks Summit/Hadoop Summit
 
Ahead Week 1 Key Slides
byaltonbaird
 
Animales en peligro de extincion
bylosdonkey
 
BNI Lake Business Builders- LOZ Vice President report
byMike Tobin
 
Setting up Your LinkedIn Account
byNET:101
 

Similar to Pattern Mining: Extracting Value from Log Data

PDF
Frequent Pattern Mining - Krishna Sridhar, Feb 2016
bySeattle DAML meetup
 
PPT
My6asso
byketan533
 
PDF
Usage and Research Challenges in the Area of Frequent Pattern in Data Mining
byIOSR Journals
 
PPT
FP growth algorithm, data mining, data analystics
byAlketaAlia
 
PPT
Tutorial on Frequent Pattern Mining Approach
byMaleehaSheikh2
 
PPT
Cs501 mining frequentpatterns
byKamal Singh Lodhi
 
PDF
Frequent Pattern Analysis, Apriori and FP Growth Algorithm
byShivarkarSandip
 
PDF
RDataMining slides-association-rule-mining-with-r
byYanchang Zhao
 
PDF
Analytical Study and Newer Approach towards Frequent Pattern Mining using Boo...
byiosrjce
 
PDF
R01732105109
byIOSR Journals
 
PPTX
Chapter 01 Introduction DM.pptx
byssuser957b41
 
PPTX
Data Mining
byNilaNila16
 
PDF
Comparative study of frequent item set in data mining
byijpla
 
PPTX
Association Rule Mining, Correlation,Clustering
byRupaRaj6
 
PDF
06FPBasic02.pdf
byAlireza418370
 
PPT
Data Mining: Concepts and Techniques_ Chapter 6: Mining Frequent Patterns, ...
bySalah Amean
 
PPTX
Association and Correlation analysis.....
byanjanasharma77573
 
PPTX
Efficient frequent pattern mining in distributed system
bySaurav Kumar
 
PDF
6 module 4
bytafosepsdfasg
 
PDF
Data mining : rule mining algorithms
byrinnocente
 
Frequent Pattern Mining - Krishna Sridhar, Feb 2016
bySeattle DAML meetup
 
My6asso
byketan533
 
Usage and Research Challenges in the Area of Frequent Pattern in Data Mining
byIOSR Journals
 
FP growth algorithm, data mining, data analystics
byAlketaAlia
 
Tutorial on Frequent Pattern Mining Approach
byMaleehaSheikh2
 
Cs501 mining frequentpatterns
byKamal Singh Lodhi
 
Frequent Pattern Analysis, Apriori and FP Growth Algorithm
byShivarkarSandip
 
RDataMining slides-association-rule-mining-with-r
byYanchang Zhao
 
Analytical Study and Newer Approach towards Frequent Pattern Mining using Boo...
byiosrjce
 
R01732105109
byIOSR Journals
 
Chapter 01 Introduction DM.pptx
byssuser957b41
 
Data Mining
byNilaNila16
 
Comparative study of frequent item set in data mining
byijpla
 
Association Rule Mining, Correlation,Clustering
byRupaRaj6
 
06FPBasic02.pdf
byAlireza418370
 
Data Mining: Concepts and Techniques_ Chapter 6: Mining Frequent Patterns, ...
bySalah Amean
 
Association and Correlation analysis.....
byanjanasharma77573
 
Efficient frequent pattern mining in distributed system
bySaurav Kumar
 
6 module 4
bytafosepsdfasg
 
Data mining : rule mining algorithms
byrinnocente
 

More from Turi, Inc.

PDF
Webinar - Patient Readmission Risk
byTuri, Inc.
 
PPTX
Webinar - Know Your Customer - Arya (20160526)
byTuri, Inc.
 
PPTX
Webinar - Product Matching - Palombo (20160428)
byTuri, Inc.
 
PPTX
Webinar - Fraud Detection - Palombo (20160428)
byTuri, Inc.
 
PPTX
Scaling Up Machine Learning: How to Benchmark GraphLab Create on Huge Datasets
byTuri, Inc.
 
PPTX
Text Analysis with Machine Learning
byTuri, Inc.
 
PPTX
Machine Learning with GraphLab Create
byTuri, Inc.
 
PPTX
Machine Learning in Production with Dato Predictive Services
byTuri, Inc.
 
PPTX
Machine Learning in 2016: Live Q&A with Carlos Guestrin
byTuri, Inc.
 
PDF
Scalable data structures for data science
byTuri, Inc.
 
PPTX
Introduction to Deep Learning for Image Analysis at Strata NYC, Sep 2015
byTuri, Inc.
 
PDF
Introduction to Recommender Systems
byTuri, Inc.
 
PDF
Machine learning in production
byTuri, Inc.
 
PPTX
Overview of Machine Learning and Feature Engineering
byTuri, Inc.
 
PPTX
SFrame
byTuri, Inc.
 
PPT
Building Personalized Data Products with Dato
byTuri, Inc.
 
PPTX
Getting Started With Dato - August 2015
byTuri, Inc.
 
PPTX
Towards a Comprehensive Machine Learning Benchmark
byTuri, Inc.
 
PDF
Dato Keynote
byTuri, Inc.
 
PDF
New Capabilities in the PyData Ecosystem
byTuri, Inc.
 
Webinar - Patient Readmission Risk
byTuri, Inc.
 
Webinar - Know Your Customer - Arya (20160526)
byTuri, Inc.
 
Webinar - Product Matching - Palombo (20160428)
byTuri, Inc.
 
Webinar - Fraud Detection - Palombo (20160428)
byTuri, Inc.
 
Scaling Up Machine Learning: How to Benchmark GraphLab Create on Huge Datasets
byTuri, Inc.
 
Text Analysis with Machine Learning
byTuri, Inc.
 
Machine Learning with GraphLab Create
byTuri, Inc.
 
Machine Learning in Production with Dato Predictive Services
byTuri, Inc.
 
Machine Learning in 2016: Live Q&A with Carlos Guestrin
byTuri, Inc.
 
Scalable data structures for data science
byTuri, Inc.
 
Introduction to Deep Learning for Image Analysis at Strata NYC, Sep 2015
byTuri, Inc.
 
Introduction to Recommender Systems
byTuri, Inc.
 
Machine learning in production
byTuri, Inc.
 
Overview of Machine Learning and Feature Engineering
byTuri, Inc.
 
SFrame
byTuri, Inc.
 
Building Personalized Data Products with Dato
byTuri, Inc.
 
Getting Started With Dato - August 2015
byTuri, Inc.
 
Towards a Comprehensive Machine Learning Benchmark
byTuri, Inc.
 
Dato Keynote
byTuri, Inc.
 
New Capabilities in the PyData Ecosystem
byTuri, Inc.
 

Recently uploaded

PPTX
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
PDF
UiPath DevConnect 2025: UiPath ScreenPlay - The Future of Human-Like Automation
byDianaGray10
 
PPTX
Closing Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
PPTX
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
PDF
Supervised Machine Learning Approaches for Log-Based Anomaly Detection: A Cas...
byMohammed BEKKOUCHE
 
PDF
Webinar: Introduction to LF Energy SEAPATH
byDanBrown980551
 
PPTX
Agentic AI presentation with Python Exercises
byParth Mane
 
PPTX
Standardising and simplifying systems and data management - Esri UK Welsh Con...
byEsri UK
 
PDF
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
PPTX
Explaining ourselves – people, computers and AI
byAlan Dix
 
PDF
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
PDF
Mutation Testing for Industrial Robotic Systems (FMAS 2025)
bySylvain Hallé
 
PPTX
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
PDF
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
PDF
November 2025 OpenMetadata Community Spotlight - Astronomer & Airflow 3
byOpenMetadata
 
PDF
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
PPTX
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
PPTX
Redcentric Data Centres: vision, mission and values
byRedcentric
 
PDF
LVM Management and Disaster Recovery.pdf
byLinuxCert Guru
 
PDF
OutSystsems User Group Brabant November 2025
bymail496323
 
Osmosis webinar presentation Nov 2025:Empowering UK Nursing Education
byJisc
 
UiPath DevConnect 2025: UiPath ScreenPlay - The Future of Human-Like Automation
byDianaGray10
 
Closing Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
Support, Monitoring, Continuous Improvement & Scaling Agentic Automation [3/3]
byUiPathCommunity
 
Supervised Machine Learning Approaches for Log-Based Anomaly Detection: A Cas...
byMohammed BEKKOUCHE
 
Webinar: Introduction to LF Energy SEAPATH
byDanBrown980551
 
Agentic AI presentation with Python Exercises
byParth Mane
 
Standardising and simplifying systems and data management - Esri UK Welsh Con...
byEsri UK
 
Infuse Intelligence Into your App with Foundry Local
byNilesh Gule
 
Explaining ourselves – people, computers and AI
byAlan Dix
 
IDSECCONF2025 - Aan Wahyu - Maze–Malware Analysis Platform.pdf
byidsecconf
 
Mutation Testing for Industrial Robotic Systems (FMAS 2025)
bySylvain Hallé
 
Neo4j Fraud GraphTalk Singapore Nov 2025
bygourisachdeva2
 
MINDCTI Revenue Release Quarter 3 2025 - Financial Presentation
byMIND CTI
 
November 2025 OpenMetadata Community Spotlight - Astronomer & Airflow 3
byOpenMetadata
 
Dev Dives: Build smarter agents with UiPath Agent Builder
byUiPathCommunity
 
Opening Plenary - Esri UK Welsh Conference 2025
byEsri UK
 
Redcentric Data Centres: vision, mission and values
byRedcentric
 
LVM Management and Disaster Recovery.pdf
byLinuxCert Guru
 
OutSystsems User Group Brabant November 2025
bymail496323
 

Pattern Mining: Extracting Value from Log Data

  • 1.
    Pattern Mining: Gettingthe most out of your log data. Krishna Sridhar Staff Data Scientist, Dato Inc. krishna_srd
  • 2.
    • Background - MachineLearning (ML) Research. - Ph.D Numerical Optimization @Wisconsin • Now - Build ML tools for data-scientists & developers @Dato. - Help deploy ML algorithms. @krishna_srd, @DatoInc About Me!
  • 3.
  • 4.
    + = Questions? • (Now)We are monitoring the chat window. • (Later) Email me srikris@dato.com. Webinars
  • 5.
  • 6.
    Creating a modelpipeline Ingest Transform Model Deploy Unstructured Data exploration data modeling Data Science Workflow Ingest Transform Model Deploy
  • 7.
    GraphLab(Create( Train Model Pipeline Deploy Models Serve Requests (REST API) Monitor Services Get Live Feedback Update Pipelines Prototype& Develop Model Pipelines Update Live Experiment Deploy New Pipeline Dato(Predic2ve(Services( Dato’s Products Dato(Distributed( We can help!
  • 8.
    Log Journey Lots ofdata Insights Profits
  • 9.
  • 10.
  • 11.
    Machine Learning inLogs Source: Mining Your Logs - Gaining Insight Through Visualization
  • 12.
  • 13.
  • 14.
  • 15.
    Frequent Pattern Mining Whatsets of items were bought together?
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
    Can we recommenditems? Rule Mining
  • 21.
  • 22.
  • 23.
    Feature Extraction 0 10 0 0 0 1 1 0 1 1 0 0 1 0 0 0 0 0 0 1 1 1 0 Receipt Space Features in Menu Space ML
  • 24.
    3 Useful DataMining Tasks Rule MiningPattern Mining Feature Extraction
  • 25.
  • 26.
    ML is nota black-box. Transparency Learning is also about understanding. Interpretability Whatever can go wrong, will go wrong. Diagnosis Moving on
  • 27.
  • 28.
    Formulating Pattern Mining Ndistinct items → 2N itemsets
  • 29.
    Formulating Pattern Mining Findthe top K most frequent sets of length at least L that occur at least M times.
  • 30.
    Formulating Pattern Mining Findthe top K most frequent sets of length at least L that occur at least M times. - max_patterns - min_length - min_support
  • 31.
    Pattern Mining N distinctitems → 2N itemsets
  • 32.
  • 33.
    Principle 1: Whatis frequent? A pattern is frequent if it occurs at least M times. {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {C, D}: 5 is frequent M = 4 {A, D}: 5 is not frequent
  • 34.
    Principle 1: Whatis frequent? A pattern is frequent if it occurs at least M times. {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {C, D}: 5 is frequent M = 4 {A, D}: 5 is not frequent min_support
  • 35.
    Principle 2: Aprioriprinciple A pattern is frequent only if a subset is frequent {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D} {B, C, D} : 5 is frequent therefore {C, D} : 5 is frequent {A} : 3 is not frequent therefore {A, D} : 3 is not frequent M = 4
  • 36.
    Two Main Algorithms •Candidate Generation - Apriori - Eclat • Pattern Growth - FP-Growth - TopK FP-Growth [GLC 1.6]
  • 37.
    Lots of Generalizations Source:http://www.philippe-fournier-viger.com/spmf/
  • 38.
    Candidate Generation Two phases 1.Candidate generation. 2. Candidate filtering. Exploit Apriori Principle!
  • 39.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 40.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 41.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 42.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 43.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 44.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : 4 {BD} : 4 {CD} : 5 {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 45.
    Candidate Generation {AB} :? {AC} : ? {AD} : ? {BC} : 4 {BD} : 4 {CD} : 5 {A} : 3 {B} : 4 {C} : 5 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {B, C, D} {A, C, D} {A, B, C, D} {A, D} {B, C, D} {B, C, D}
  • 46.
    Pattern Growth Two phases 1.Candidate filtering 2. Conditional database constructions. Avoid full scans over the data & large candidate sets!
  • 47.
    Pattern Growth -Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : 1 {AC} : 2 {AD} : 3 {BD} : 4 {CD} : 4 {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : 0 {ABD} : 1 {ACD} : 2 {BCD} : 2 {BC} : 2
  • 48.
    Pattern Growth -Preprocessing {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6
  • 49.
    Pattern Growth -Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : ? {B} : ? {C} : ? {D} : ? { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
  • 50.
    Pattern Growth -Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
  • 51.
    Pattern Growth -Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : ? {AC} : ? {AD} : ? {BD} : ? {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : ?
  • 52.
    Pattern Growth -Depth First {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : 2
  • 53.
    Pattern Growth {B} :4 { } : 6 Call: Growth(db = DB{}, item = B, freq = {B,C,D}) DB{} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D}
  • 54.
    Pattern Growth {B} :4 { } : 6 Conditional Database Construction DB{} DB{B} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} {C, D} {D} {C, D} {D}
  • 55.
    Pattern Growth {B} :4 { } : 6 Candidate Filtering DB{B} {C, D} {D} {C, D} {D} {D} : 4 {C} : 2 DB{} {B, C, D} {A, C, D} {B, D} {A, C, D} {B, C, D} {A, B, D} DB{B} Add {BD} as frequent
  • 56.
    Pattern Growth -Depth First {C, D} {D} {C, D} {D} {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : ? {ACD} : ? {BCD} : ? {BC} : 2
  • 57.
    Pattern Growth Recurse: Growth(db= DB{B}, item = D, freq = {D}) DB{B} {C, D} {D} {C, D} {D} {B} : 4 { } : 6 {BD} : 4 DB{BD}
  • 58.
    Pattern Growth -Depth First {AB} : X {AC} : ? {AD} : ? {BD} : 4 {CD} : ? {A} : 3 {B} : 4 {C} : 4 {D} : 6 { } : 6 {ABC} : ? {ABD} : X {ACD} : ? {BCD} : X {BC} : 2
  • 59.
    Compare & Constrast •Candidate Generation + Better than brute force + Filters candidate sets - Multiple passes over the data • Pattern Growth + Fewer passes over the data + Space efficient.
  • 60.
    Compare & Constrast •Candidate Generation + Better than brute force + Filters candidate sets - Multiple passes over the data • Pattern Growth + Fewer passes over the data + Space efficient. Better choice
  • 61.
    FP-Tree Compression Figures FromFlorian Verhein’s Slides on FP-Growth
  • 62.
    FP-Growth Algorithm Figures FromFlorian Verhein’s Slides on FP-Growth Two phases 1. Candidate filtering. 2. Conditional database constructions.
  • 63.
    TopK FP-Growth Algorithm Similarto FP-Growth 1. Dynamically raise min_support. 2. Estimates of min_support greatly help.
  • 64.
    Performance on WebsiteLogs • 1.5m events • 84k sessions • 3k unique ids
  • 65.
  • 66.
  • 67.
    Bags + Sequences ×2 Itemset: {Item} Bags: {Item: quantity} Sequences : (item)
  • 68.
  • 69.
    Creating a modelpipeline Ingest Transform Model Deploy Unstructured Data exploration data modeling Data Science Workflow Ingest Transform Model Deploy
  • 70.
  • 71.
    Summary Log Data Mining ≠ RocketScience • FP-Growth for finding frequent patterns. • Find rules from patterns to make predictions. • Extract features for useful ML in pattern space.
  • 72.
    SELECT questions FROMaudience WHERE difficulty == “Easy” Thanks!
  • 73.