Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark

CRYSTAL BALL EVENT PREDICTION (MAPREDUCE)
& LOG ANALYSIS (SPARK)
By: Jivan Nepali, 985095 Big Data (CS522) Project

PRESENTATION OVERVIEW
Pair Approach
• Pseudo-code for Pair Approach
• Java Implementation for Pair Approach
• Pair Approach Result
Stripe Approach
• Pseudo-code for Stripe Approach
• Java Implementation for Stripe Approach
• Stripe Approach Result
Hybrid Approach
• Pseudo-code for Hybrid Approach
• Java Implementation for Hybrid Approach
• Hybrid Approach Result
• Comparison of three Approaches
Spark
• LogAnalysis – Problem Description
• LogAnalysis – Expected Outcomes
• LogAnalysis – Scala Implementation
• LogAnalysis – Results

PSEUDO CODE – MAPPER
Class MAPPER
method INITIALIZE
H = new Associative Array
method MAP (docid a, doc d)
for all term w in doc d do
for all term u in Neighbors(w) do
H { Pair (w, u) } = H {Pair (w, u) } + count 1 // Tally counts
H { Pair(w, *) } = H { Pair (w, *) } + count 1 // Tally counts for *
method CLOSE
for all Pair (w, u) in H do
EMIT ( Pair (w, u), count H { Pair (w, u) } )

PSUDEO CODE - REDUCER
Class REDUCER
method INITIALIZE
TOTALFREQ = 0
method REDUCE (Pair p, counts [c1, c2, c3, … ])
sum = 0
for all count c in counts [c1, c2, c3, … ]) do
sum = sum + c
if ( p.getNeighbor() == “*”)) then //Neighbor is second element of the pair
TOTALFREQ = sum
else
EMIT ( Pair p, sum / TOTALFREQ)

IMPLEMENTATION – MAPPER CONTD…

PAIR APPROACH – MAP INPUT RECORDS
18 34 56 29 12 34 56 92 29 34 12
92 29 18 12 34 79 29 56 12 34 18

STRIPE APPROACH
IMPLEMENTATION

Class MAPPER
method INITIALIZE
S = H { w } // Initialize a new Associative Array if H {w} is NULL
S { u } = S { u } + count 1 // Tally counts
H { w } = S
method CLOSE
for all term t in H do
EMIT ( term t, stripe H { t } )

Class REDUCER
method INITIALIZE
TOTALFREQ = 0
Hf = new Associative Array
method REDUCE (term t, stripes [H1, H2, H3, … ])
for all stripe H in stripes [H1, H2, H3, … ]) do
for all term w in stripe H do
Hf { w } = Hf { w } + H { w } // Hf = Hf + H ; Element-wise addition
TOTALFREQ = TOTALFREQ + count H { w }
for all term w in stripe Hf do
Hf { w } = Hf { w } / TOTALFREQ
EMIT ( term t, stripe Hf )

IMPLEMENTATION – REDUCER CONTD…

STRIPE APPROACH – MAP INPUT RECORDS
18 34 56 29 12 34 56 92 29 34 12
92 29 18 12 34 79 29 56 12 34 18

HYBRID APPROACH
IMPLEMENTATION

Class MAPPER
method INITIALIZE
H { Pair (w, u) } = H {Pair (w, u) } + count 1 // Tally counts
method CLOSE
for all Pair (w, u) in H do
EMIT ( Pair (w, u), count H { Pair (w, u) } )

Class REDUCER
method INITIALIZE
TOTALFREQ = 0
PREVKEY = “”
method REDUCE (Pair p, counts [C1, C2, C3, … ])
sum = 0
for all count c in counts [ C1, C2, C3, … ] do
sum = sum + c
if ( PREVKEY <> p.getKey( )) then
EMIT ( PREVKEY, Hf / TOTALFREQ ) // Element-wise divide
TOTALFREQ = 0

PSUDEO CODE – REDUCER CONTD…
TOTALFREQ = TOTALFREQ + sum
Hf { p.getNeighbor( ) } = Hf { p.getNeighbor( ) } + sum
PREVKEY = p.getKey( )
method CLOSE // for the remaining last key
EMIT ( PREVKEY, Hf / TOTALFREQ ) // Element-wise divide

IMPLEMENTATION – MAPPER CONTED…

IMPLEMENTATION – REDUCER CONTD …

HYBRID APPROACH – MAP INPUT RECORDS
18 34 56 29 12 34 56 92 29 34 12
92 29 18 12 34 79 29 56 12 34 18

MAP-REDUCE JOB PERFORMANCE
COMPARISON WITH COUNTERS
Description Pair Approach Stripe Approach Hybrid Approach
Map Input Records 2 2 2
Map Output Records 47 7 40
Map Output Bytes 463 416 400
Map Output Materialized Bytes 563 436 486
Input-split Bytes 147 149 149
Combine Input Records 0 0 0
Combine Output Records 0 0 0
Reduce Input Groups 47 7 40
Reduce Shuffle Bytes 563 436 486
Reduce Input Records 47 7 40
Reduce Output Records 40 7 7
Shuffled Maps 1 1 1
GC Time Elapsed (ms) 140 175 129
CPU Time Spent (ms) 1540 1530 1700
Physical Memory (bytes) Snapshot 357101568 354013184 352686080
Virtual Memory (bytes) Snapshot 3022008320 3019862016 3020025856
Total Committed Heap Usage (bytes) 226365440 226365440 226365440

LOG ANALYSIS
• Log data is a definitive record of what's
happening in every business, organization
or agency and it’s often an untapped
resource when it comes to troubleshooting
and supporting broader business
objectives.
• 1.5 Millions Log Lines Per Second !

PROBLEM DESCRIPTION
• Web-access log data from Splunk
• Three log files ( ~ 12 MB)
Features
• Extract top selling products
• Extract top selling product categories
• Extract top client IPs visiting the e-commerce site
Sample Data

SPARK, SCALA CONFIGURATION IN ECLIPSE
• Download Scala IDE from http://scala-ide.org/download/sdk.html for Linux 64 bit

SPARK, SCALA CONFIGURATION IN ECLIPSE
• Open the Scala IDE
• Create a new Maven Project
• Configure the pom.xml file
• maven clean, maven install
• Set the Scala Installation to Scala 2.10.4
from Project -> Scala -> Set Installation

LOG ANALYSIS - SCALA IMPLEMENTATION
• Add new Scala Object
to the src directory of
the project

LOG ANALYSIS - SCALA IMPLEMENTATION

CREATING & EXECUTING THE .JAR FILE
• Open Linux Terminal
• Go to the project directory & Perform mvn clean, mvn package to create the .JAR
file
• Change the permission of .jar as executable ( sudo chmod 777 filename.jar )
• Run the .jar file by providing the input and output directories as arguments
spark-submit --class cs522.sparkproject.LogAnalyzer $LOCAL_DIR/spark/sparkproject-
0.0.1-SNAPSHOT.jar $HDFS_DIR/spark/input $HDFS_DIR/spark/output

LOG ANALYSIS – RESULT (TOP PRODUCT IDs)

LOG ANALYSIS – RESULT (TOP PRODUCT CATEGORIES)

LOG ANALYSIS – RESULT (TOP CLIENT IPs)

Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark

More Related Content

What's hot

Viewers also liked

Similar to Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark

More from Jivan Nepali

Recently uploaded

Crystal Ball Event Prediction and Log Analysis with Hadoop MapReduce and Spark