1.pptx .

Mining Frequent Patterns,
Associations, and Correlations:
Basic Concepts and Methods

Introduction
 Imagine that you are a sales manager at AllElectronics, and you are talking to a
customer who recently bought a PC and a digital camera from the store.
 What should you recommend to her next?
 Information about which products are frequently purchased by your customers
following their purchases of a PC and a digital camera in sequence would be
very helpful in making your recommendation.

 Frequent patterns and association rules are the knowledge that you want to
mine in such a scenario.
 Frequent patterns are patterns that appear frequently in a data set.
 For example, a set of items, such as milk and bread, that appear frequently
together in a transaction data set is a frequent itemset.
 A subsequence, such as buying first a PC, then a digital camera, and then a
memory card, if it occurs frequently in a shopping history database, is a
(frequent) sequential pattern.
 A substructure can refer to different structural forms, such as subgraphs,
subtrees, which may be combined with itemsets or subsequences.

 If a substructure occurs frequently, it is called a (frequent) structured pattern.
 Finding frequent patterns plays an essential role in mining associations,
correlations, and many other interesting relationships among data.
 Moreover, it helps in data classification, clustering, and other data mining tasks.
 Thus, frequent pattern mining has become an important data mining task and
a focused theme in data mining research.

 Basic Concepts
 Frequent pattern mining searches for recurring relationships in a given data set.
 Market Basket Analysis: A Motivating Example
 Frequent itemset mining leads to the discovery of associations and correlations
among items in large transactional or relational data sets.
 With massive amounts of data continuously being collected and stored, many
industries are becoming interested in mining such patterns from their
databases.
 The discovery of interesting correlation relationships among huge amounts of
business transaction records can help in many business decision-making
processes such as catalog design, cross-marketing, and customer shopping
behavior analysis.

 A typical example of frequent itemset mining is market basket analysis.
 This process analyzes customer buying habits by finding associations between
the different items that customers place in their “shopping baskets”.

 The discovery of these associations can help retailers develop marketing
strategies by gaining insight into which items are frequently purchased together
by customers.
 For instance, if customers are buying milk, how likely are they to also buy bread
(and what kind of bread) on the same trip to the supermarket?
 This information can lead to increased sales by helping retailers do selective
marketing and plan their shelf space.
 Let’s look at an example of how market basket analysis can be useful.

 Example 6.1 Market basket analysis.
 Suppose, as manager of an AllElectronics branch, you would like to learn more
about the buying habits of your customers.
 Specifically, you wonder, “Which groups or sets of items are customers likely to
purchase on a given trip to the store?”
 To answer your question, market basket analysis may be performed on the retail
data of customer transactions at your store.
 You can then use the results to plan marketing or advertising strategies, or in the
design of a new catalog.

 For instance, market basket analysis may help you design different store
layouts.
 In one strategy, items that are frequently purchased together can be placed
in proximity to further encourage the combined sale of such items.
 If customers who purchase computers also tend to buy antivirus software at the
same time, then placing the hardware display close to the software display
may help increase the sales of both items.
 In an alternative strategy, placing hardware and software at opposite ends of
the store may attract customers who purchase such items to pick up other
items along the way.
 For instance, after deciding on an expensive computer, a customer may
observe security systems for sale while heading toward the software display to
purchase antivirus software, and may decide to purchase a home security
system as well.

 Market basket analysis can also help retailers plan which items to put on
sale at reduced prices.
 If customers tend to purchase computers and printers together, then
having a sale on printers may encourage the sale of printers as well as
computers.

 Many Business enterprises accumulate large quantity of data from their day-to-
day operations, commonly known as Market-basket transactions.
 Each row in this table corresponds to a transaction, which contains a unique
identifier labelled TID and a set of items bought by a given customer, which is of
much interest for learning the purchasing behavior of the customers.
TID Items
1 {Bread, Milk}
2 {Bread, Diapers, Beer, Eggs}
3 {Milk, Diapers, Beer, Cola}
4 {Bread, Milk, Diapers, Beer}
5 {Bread, Milk, Diapers, Cola}

 Market basket data can be represented in a binary format where each row
corresponds to a transaction and each column corresponds to an item.
 An item can be treated as a binary variable whose value is one if the item is
present in a transaction and zero otherwise.
 The Boolean vectors can be analyzed for buying patterns that reflect items that
are frequently associated or purchased together.
TID Bread Milk Diapers Beer Eggs Cola
1 1 1 0 0 0 0
2 1 0 1 1 1 1
3 0 1 1 1 0 0
4 1 1 1 1 0 0

 Item set
 In association analysis, a collection of zero or more items is termed as item set
 E.g., {Beer, Diapers, Milk} is an example of a 3-itemset
 The null set is an itemset that does not contain any items
 Transaction Width
 Number of items present in a transaction

 Association Rule
 An Association Rule is an implication expression of the form X → Y, where X and
Y are disjoint itemsets.
 The strength of an association rule can be measured in terms of its support and
confidence.
 Support
 Support determines how often a rule is applicable to a given data set. It is the
percentage of transactions in D that contain X Y.
∪
S(X → Y) = σ (X Y)/ N
∪

 Confidence
 Confidence determines how frequently items in Y appears in transactions that
contain X.
 It is the percentage of transactions containing X that also contain Y.
C(X → Y) = σ (X Y)/
∪ σ(X)


 Consider the rule {Milk, Diapers} → {Beer}
 Support count for {Milk, Diapers, Beer} is 2 and total number of transactions is 5,
Support = 2/5 = 0.4 (40%)
 Confidence is obtained by dividing the support count for {Milk, Diapers , Beer}
by the support count for {Milk, Diapers}
 Since there are three transactions that contains milk and diapers,
Confidence = 2/ 3= 0.67 (67%)

 Given the definitions of transactions T, the goal of association mining is to find
all rules having
 support ≥ minsup threshold
 Confidence ≥ minconf threshold
 In general, association rule mining can be viewed as a two-step process:
1. Frequent Itemset Generation- Find all frequent itemsets, by definition, each of
these itemsets will occur at least as frequently as a predetermined minimum
support count, minsup threshold.
2. Rule Generation- Generate strong association rules from the frequent
itemsets, by definition, these rules must satisfy minimum support and minimum
confidence.

 Let I = {I1, I2,..., Im} be an itemset.
 Let D, be a set of database transactions where each transaction T is a nonempty
itemset such that T I.
⊆
 Each transaction is associated with an identifier, called a TID.
 Let A be a set of items.
 A transaction T is said to contain A if A T.
⊆
 An association rule is an implication of the form A B, where A I, B I, A ≠ ,
⇒ ⊂ ⊂ ∅
B ≠ , and A ∩B = φ
∅ .
 The rule A B holds in the transaction set D with support s, where s is the
⇒
percentage of transactions in D that contain A B (i.e., the union of sets A and B
∪
say, or, both A and B).
 This is taken to be the probability, P(A B).
∪
 The rule A B has confidence c in the transaction set D, where c is the
⇒
percentage of transactions in D containing A that also contain B.

 An itemset X is closed in a data set D if there exists no proper super-itemset Y
(Y is a proper super-itemset of X if X is a proper sub-itemset of Y, that is, if X Y.
⊂
In other words, every item of X is contained in Y but there is at least one item of
Y that is not in X) such that Y has the same support count as X in D.
 An itemset X is a closed frequent itemset in set D if X is both closed and
frequent in D.
 An itemset X is a maximal frequent itemset (or max-itemset) in a data set D if X is
frequent, and there exists no super-itemset Y such that X Y and Y is frequent in
⊂
D.

 Frequent Itemset Mining Methods
 Apriori Algorithm: Finding Frequent Itemsets by Confined Candidate Generation
 Apriori is a important algorithm proposed by R. Agrawal and R. Srikant in 1994 for
mining frequent itemsets for Boolean association rule.
 The name of the algorithm is based on the fact that the algorithm uses prior
knowledge of frequent itemset properties.
 Apriori employs an iterative approach known as a level-wise search, where k-
itemsets are used to explore (k + 1)-itemsets.
 First, the set of frequent 1-itemsets is found by scanning the database to accumulate
the count for each item, and collecting those items that satisfy minimum support.
 The resulting set is denoted by L1.
 Next, L1 is used to find L2, the set of frequent 2-itemsets, which is used to find L3, and so
on, until no more frequent k-itemsets can be found.
 The finding of each Lk requires one full scan of the database.

 Apriori property:
 All nonempty subsets of a frequent itemset must also be frequent.
 The Apriori property is based on the following observation.
 By definition, if an itemset I does not satisfy the minimum support threshold,
min sup, then I is not frequent, that is, P(I) < min sup.
 If an item A is added to the itemset I, then the resulting itemset (i.e., I A)
∪
cannot occur more frequently than I.
 Therefore, I A is not frequent either, that is, P(I A) < min sup.
∪ ∪
 “How is the Apriori property used in the algorithm?”
 To understand this, let us look at how Lk−1 is used to find Lk for k ≥ 2.
 A two-step process is followed, consisting of join and prune actions.

1. The join step: To find Lk , a set of candidate k-itemsets is generated by joining
Lk−1 with itself.
 This set of candidates is denoted Ck .
 Let l1 and l2 be itemsets in Lk−1.
 The notation li[j] refers to the jth item in li (e.g., l1[k − 2] refers to the second to the
last item in l1).
 For efficient implementation, Apriori assumes that items within a transaction or
itemset are sorted in lexicographic order.
 For the (k − 1)-itemset, li , this means that the items are sorted such that li[1] < li[2]
< ··· < li[k − 1].
 The join, Lk−1 ⨝ Lk−1, is performed, where members of Lk−1 are joinable if their first (k
− 2) items are in common.

 That is, members l1 and l2 of Lk−1 are joined if (l1[1] = l2[1]) (l
∧ 1[2] = l2[2]) ···
∧ ∧
(l1[k − 2] = l2[k − 2]) (l
∧ 1[k − 1] < l2[k − 1]).
 The condition l1[k − 1] < l2[k − 1] simply ensures that no duplicates are
generated. The resulting itemset formed by joining l1 and l2 is {l1[1], l1[2],..., l1[k
− 2], l1[k − 1], l2[k − 1]}.

2. The prune step: Ck is a superset of Lk , that is, its members may or may not be
frequent, but all of the frequent k-itemsets are included in Ck .
 A database scan to determine the count of each candidate in Ck would result in the
determination of Lk (i.e., all candidates having a count no less than the minimum
support count are frequent by definition, and therefore belong to Lk).
 Ck , however, can be huge, and so this could involve heavy computation.
 To reduce the size of Ck , the Apriori property is used as follows.
 Any (k − 1)-itemset that is not frequent cannot be a subset of a frequent k-itemset.
 Hence, if any (k − 1)-subset of a candidate k-itemset is not in Lk−1, then the candidate
cannot be frequent either and so can be removed from Ck .
 This subset testing can be done quickly by maintaining a hash tree of all frequent
itemsets.

 Apriori Example . Let’s look at a concrete example, based on the
AllElectronics transaction database, D, of table below
 There are nine transactions in this database, that is, |D| = 9.

 We use Figure below to illustrate the Apriori algorithm for finding frequent
itemsets in D.

1. In the first iteration of the algorithm, each item is a member of the set of
candidate 1-itemsets, C1. The algorithm simply scans all of the transactions to
count the number of occurrences of each item.
2. Suppose that the minimum support count required is 2, that is, min sup = 2. (Here,
we are referring to absolute support because we are using a support count. The
corresponding relative support is 2/9 = 22%.) The set of frequent 1-itemsets, L1, can
then be determined. It consists of the candidate 1-itemsets satisfying minimum
support. In our example, all of the candidates in C1 satisfy minimum support.
3. To discover the set of frequent 2-itemsets, L2, the algorithm uses the join L1 ⨝
L1 to generate a candidate set of 2-itemsets, C2. C2 consists of 2-itemsets. Note
that no candidates are removed from C2 during the prune step because each
subset of the candidates is also frequent.
4. Next, the transactions in D are scanned and the support count of each
candidate itemset in C2 is accumulated, as shown in the middle table of the
second row in Figure.

5. The set of frequent 2-itemsets, L2, is then determined, consisting of those
candidate 2-itemsets in C2 having minimum support.
6. The generation of the set of the candidate 3-itemsets, C3, is detailed below.

From the join step, we first get C3 = L2 ⨝ L2 = {{I1, I2, I3}, {I1, I2, I5}, {I1, I3, I5}, {I2,
I3, I4}, {I2, I3, I5}, {I2, I4, I5}}. Based on the Apriori property that all subsets of a
frequent itemset must also be frequent, we can determine that the four latter
candidates cannot possibly be frequent. We therefore remove them from C3,
thereby saving the effort of unnecessarily obtaining their counts during the
subsequent scan of D to determine L3. Note that when given a candidate k-
itemset, we only need to check if its (k − 1)-subsets are frequent since the
Apriori algorithm uses a level-wise search strategy. The resulting pruned version
of C3 is shown in the first table of the bottom row of Figure,
7. The transactions in D are scanned to determine L3, consisting of those
candidate 3-itemsets in C3 having minimum support.
8. The algorithm uses L3 ⨝ L3 to generate a candidate set of 4-itemsets, C4.
Although the join results in {{I1, I2, I3, I5}}, itemset {I1, I2, I3, I5} is pruned because
its subset {I2, I3, I5} is not frequent. Thus, C4 = φ, and the algorithm terminates,
having found all of the frequent itemsets.

 Figure below shows pseudocode for the Apriori algorithm and its related
procedures.

 Step 1 of Apriori finds the frequent 1-itemsets, L1.
 In steps 2 through 10, Lk−1 is used to generate candidates Ck to find Lk for k ≥ 2.
 The apriori gen procedure generates the candidates and then uses the
Apriori property to eliminate those having a subset that is not frequent (step
3).
 Once all of the candidates have been generated, the database is scanned
(step 4).
 For each transaction, a subset function is used to find all subsets of the
transaction that are candidates (step 5), and the count for each of these
candidates is accumulated (steps 6 and 7).
 Finally, all the candidates satisfying the minimum support (step 9) form the set
of frequent itemsets, L (step 11).

 A procedure can then be called to generate association rules from the
frequent itemsets.
 The apriori gen procedure performs two kinds of actions, namely, join and
prune.
 In the join component, Lk−1 is joined with Lk−1 to generate potential
candidates (steps 1–4).
 The prune component (steps 5–7) employs the Apriori property to remove
candidates that have a subset that is not frequent.
 The test for infrequent subsets is shown in procedure has infrequent subset.

 Example-
 Assume the support threshold to be 60%, which is equivalent to a minimum
support count equal to 3
 Initially, every item is considered as a candidate 1-itemset
TID Items
1 Bread, Milk
2 Bread, Diaper, Beer, Eggs
3 Milk, Diaper, Beer, Coke
4 Bread, Milk, Diaper, Beer
5 Bread, Milk, Diaper, Coke

Item Count
Bread 4
Coke 2
Milk 4
Beer 3
Diaper 4
Eggs 1
Itemset Count
{Bread,Milk} 3
{Bread,Beer} 2
{Bread,Diaper} 3
{Milk,Beer} 2
{Milk,Diaper} 3
{Beer,Diaper} 3
Itemset Count
{Bread,Milk,Diaper} 3
Items (1-itemsets)
Pairs (2-itemsets)
(No need to generate
candidates involving Coke
or Eggs)
Triplets (3-itemsets)
Minimum Support = 3

 After counting their supports, the candidate itemsets {Cola} and {Eggs} are
discarded because they appear in fewer than three transactions
 In the next iteration, candidate 2-itemsets are generated using only the frequent 1-
itemsets because the Apriori principle ensures that all supersets of the infrequent 1-
itemsets must be infrequent
 Because there are only four frequent 1-itemsets, the number of candidate 2-
itemsets generated by the algorithm is (4
2 ) = 6
 Two of these six candidates {Beer, Bread} and {Beer, Milk} are subsequently found
to be infrequent after computing their support values
 The remaining four candidates are frequent, and thus will be used to generate
candidate 3-itemsets
 With the Apriori principle, we need only to keep candidate 3-itemsets whose
subsets are frequent
 The only candidate that has this property is {Bread, Diapers, Milk}

 The effectiveness of Apriori pruning strategy can be shown by counting the
number of candidate itemsets generated
 With Apriori principle, the number of candidate itemset generated is
computed as
(6
1) + (4
2) + 1 = 6 + 6 + 1 = 13 candidates

 Generating Association Rules from Frequent Itemsets
 Once the frequent itemsets from transactions in a database D have been
found, it is straightforward to generate strong association rules from them (where
strong association rules satisfy both minimum support and minimum
confidence).
 This can be done using Eq. for confidence
 The conditional probability is expressed in terms of itemset support count, where
support count(A B) is the number of transactions containing the itemsets A B,
∪ ∪
and support count(A) is the number of transactions containing the itemset A.

 Based on this equation, association rules can be generated as follows:
 Because the rules are generated from frequent itemsets, each one
automatically satisfies the minimum support.
 Frequent itemsets can be stored ahead of time in hash tables along with their
counts so that they can be accessed quickly.

 Example -Generating association rules.
 Let’s try an example based on the transactional data for AllElectronics shown
before in Table 6.1. The data contain frequent itemset X = {I1, I2, I5}.
 What are the association rules that can be generated from X?
 The nonempty subsets of X are {I1, I2}, {I1, I5}, {I2, I5}, {I1}, {I2}, and {I5}.
 The resulting association rules are as shown below, each listed with its
confidence:
 If the minimum confidence threshold is, say, 70%, then only the second, third,
and last rules are output, because these are the only ones generated that are
strong.

 Improving the Efficiency of Apriori
 “How can we further improve the efficiency of Apriori-based mining?”
 Many variations of the Apriori algorithm have been proposed that focus on
improving the efficiency of the original algorithm.
 Several of these variations are summarized as follows:

 Hash-based technique (hashing itemsets into corresponding buckets):
 A hash-based technique can be used to reduce the size of the candidate k-
itemsets, Ck , for k > 1.
 Hash table is used as data structure.
 First iteration is required to count the support of each element.
 From second iteration, efforts are made to enhance execution of Apriori by
utilizing hash table concept
 Hash table minimizes the number of item set generated in second iteration.
 In second iteration, i.e. 2-item set generation, for every combination of two
items, we map them to diverse bucket of hash table structure and increment
the bucket count.
 If count of bucket is less than min.sup_count, we remove them from candidate
sets.

 A 2-itemset with a corresponding bucket count in the hash table that is below
the support threshold cannot be frequent and thus should be removed from
the candidate set.
 Such a hash-based technique may substantially reduce the number of
candidate k-itemsets examined (especially when k = 2).

 Partitioning (partitioning the data to find candidate itemsets):
 A partitioning technique can be used that requires just two database scans to
mine the frequent itemsets (Figure below).

 It consists of two phases. In phase I, the algorithm divides the transactions of D into n
non-overlapping partitions.
 If the minimum relative support threshold for transactions in D is min sup, then the
minimum support count for a partition is min sup × the number of transactions in that
partition.
 Example- consider the transactions in the Boolean form.
I1 I2 I3 I4 I5
T1 1 0 0 0 1
T2 0 1 0 1 0
T3 0 0 0 1 1
T4 0 1 1 0 0
T5 0 0 0 0 1
T6 0 1 1 1 0

 Database is divided into three partitions.,each having two transactions with a support
of 20%
 the support count is 1 for local i.e 20% of 2 transactions is 1.
 The support count is 3 for global ie. 20% of 6 is 2.

 For each partition, all the local frequent itemsets (i.e., the itemsets frequent
within the partition) are found.
 A local frequent itemset may or may not be frequent with respect to the entire
database, D.
 However, any itemset that is potentially frequent with respect to D must occur
as a frequent itemset in at least one of the partitions.
 Therefore, all local frequent itemsets are candidate itemsets with respect to D.
 The collection of frequent itemsets from all partitions forms the global candidate
itemsets with respect to D.
 In phase II, a second scan of D is conducted in which the actual support of
each candidate is assessed to determine the global frequent itemsets.
 Partition size and the number of partitions are set so that each partition can fit
into main memory and therefore be read only once in each phase.

 A Pattern-Growth Approach for Mining Frequent Itemsets
 In many cases the Apriori candidate generate-and-test method significantly
reduces the size of candidate sets, leading to good performance gain.
 However, it can suffer from two nontrivial costs:
 It may still need to generate a huge number of candidate sets. For
example, if there are 104
frequent 1-itemsets, the Apriori algorithm will
need to generate more than 107
candidate 2-itemsets.
 It may need to repeatedly scan the whole database and check a large
set of candidates by pattern matching. It is costly to go over each
transaction in the database to determine the support of the candidate
itemsets.

 “Can we design a method that mines the complete set of frequent itemsets
without such a costly candidate generation process?”
 An interesting method in this attempt is called frequent pattern growth, or
simply FP-growth, which adopts a divide-and-conquer strategy as follows.
 First, it compresses the database representing frequent items into a frequent
pattern tree, or FP-tree, which retains the itemset association information.
 It then divides the compressed database into a set of conditional
databases, each associated with one frequent item or “pattern fragment,”
and mines each database separately.
 For each “pattern fragment,” only its associated data sets need to be
examined.
 Therefore, this approach may substantially reduce the size of the data sets to
be searched, along with the “growth” of patterns being examined.

 Example -FP-growth (finding frequent itemsets without candidate generation).
 We reexamine the mining of transaction database, D, of Table below using the
frequent pattern growth approach.
 The first scan of the database is the same as Apriori, which derives the set of
frequent items (1-itemsets) and their support counts (frequencies).
 Let the minimum support count be 2.
 The set of frequent items is sorted in the order of descending support count.
 This resulting set or list is denoted by L.
 Thus, we have L = {{I2: 7}, {I1: 6}, {I3: 6}, {I4: 2}, {I5: 2}}.

 An FP-tree is then constructed as follows.
 First, create the root of the tree, labeled with “null.”
 Scan database D a second time.
 The items in each transaction are processed in L order (i.e., sorted according
to descending support count), and a branch is created for each transaction.
 For example, the scan of the first transaction, “T100: I1, I2, I5,” which contains
three items (I2, I1, I5 in L order), leads to the construction of the first branch of
the tree with three nodes, <I2: 1>, <I1: 1>, and <I5: 1>, where I2 is linked as a
child to the root, I1 is linked to I2, and I5 is linked to I1.
 The second transaction, T200, contains the items I2 and I4 in L order, which
would result in a branch where I2 is linked to the root and I4 is linked to I2.

 However, this branch would share a common prefix, I2, with the existing path for
T100.
 Therefore, we instead increment the count of the I2 node by 1, and create a
new node, <I4: 1>, which is linked as a child to <I2: 2>.
 In general, when considering the branch to be added for a transaction, the
count of each node along a common prefix is incremented by 1, and nodes for
the items following the prefix are created and linked accordingly.
 To facilitate tree traversal, an item header table is built so that each item points
to its occurrences in the tree via a chain of node-links.

 The tree obtained after scanning all the transactions is shown in Figure below
with the associated node-links.
 In this way, the problem of mining frequent patterns in databases is
transformed into that of mining the FP-tree.

 The FP-tree is mined as follows.
 Start from each frequent length-1 pattern (as an initial suffix pattern), construct
its conditional pattern base (a “sub-database,” which consists of the set of
prefix paths in the FP-tree co-occurring with the suffix pattern), then construct
its (conditional) FP-tree, and perform mining recursively on the tree.
 The pattern growth is achieved by the concatenation of the suffix pattern with
the frequent patterns generated from a conditional FP-tree.

 Mining of the FP-tree is summarized in Table below and detailed as follows
 We first consider I5, which is the last item in L, rather than the first.
 The reason for starting at the end of the list will become apparent as we
explain the FP-tree mining process. I5 occurs in two FP-tree branches of
Figure(previous slide).
 (The occurrences of I5 can easily be found by following its chain of node-
links.)
 The paths formed by these branches are <I2, I1, I5: 1> and <I2, I1, I3, I5: 1>.

 Therefore, considering I5 as a suffix, its corresponding two prefix paths are <I2,
I1: 1> and <I2, I1, I3: 1>, which form its conditional pattern base.
 Using this conditional pattern base as a transaction database, we build an I5-
conditional FP-tree, which contains only a single path, <I2: 2, I1: 2>; I3 is not
included because its support count of 1 is less than the minimum support
count.
 The single path generates all the combinations of frequent patterns: {I2, I5: 2},
{I1, I5: 2}, {I2, I1, I5: 2}.
 For I4, its two prefix paths form the conditional pattern base, {{I2 I1: 1}, {I2: 1}},
which generates a single-node conditional FP-tree, <I2: 2>, and derives one
frequent pattern, {I2, I4: 2}.
 Similar to the preceding analysis, I3’s conditional pattern base is {{I2, I1: 2}, {I2:
2}, {I1: 2}}.

 Its conditional FP-tree has two branches, <I2: 4, I1: 2> and <I1: 2>, as shown in
Figure below, which generates the set of patterns {{I2, I3: 4}, {I1, I3: 4}, {I2, I1, I3:
2}}.
 Finally, I1’s conditional pattern base is {{I2: 4}}, with an FP-tree that contains only
one node, <I2: 4>, which generates one frequent pattern, {I2, I1: 4}.

 This mining process is summarized in Figure below.

 The FP-growth method transforms the problem of finding long frequent patterns
into searching for shorter ones in much smaller conditional databases recursively
and then concatenating the suffix.
 It uses the least frequent items as a suffix, offering good selectivity.
 The method substantially reduces the search costs.
 When the database is large, it is sometimes unrealistic to construct a main
memory-based FP-tree.
 A study of the FP-growth method performance shows that it is efficient and
scalable for mining both long and short frequent patterns, and is about an order
of magnitude faster than the Apriori algorithm.

 Mining Frequent Itemsets Using the Vertical Data Format
 Both the Apriori and FP-growth methods mine frequent patterns from a set of
transactions in TID-itemset format (i.e., {TID : itemset}), where TID is a transaction
ID and itemset is the set of items bought in transaction TID.
 This is known as the horizontal data format.
 Alternatively, data can be presented in item-TID set format (i.e., {item : TID set}),
where item is an item name, and TID set is the set of transaction identifiers
containing the item.
 This is known as the vertical data format.

 Example -Mining frequent itemsets using the vertical data format.
 Consider the horizontal data format of the transaction database, D, of Table
below

 This can be transformed into the vertical data format shown in Table below by
scanning the data set once.
 Mining can be performed on this data set by intersecting the TID sets of every
pair of frequent single items.
 The minimum support count is 2.

 Because every single item is frequent in Table above, there are 10 intersections
performed in total, which lead to eight nonempty 2-itemsets, as shown in Table
below.
 Notice that because the itemsets {I1, I4} and {I3, I5} each contain only one
transaction, they do not belong to the set of frequent 2-itemsets.

 Based on the Apriori property, a given 3-itemset is a candidate 3-itemset only if
every one of its 2-itemset subsets is frequent.
 The candidate generation process here will generate only two 3-itemsets: {I1,
I2, I3} and {I1, I2, I5}.
 By intersecting the TID sets of any two corresponding 2-itemsets of these
candidate 3-itemsets, it derives Table below, where there are only two frequent
3-itemsets: {I1, I2, I3: 2} and {I1, I2, I5: 2}.

 This example illustrates the process of mining frequent itemsets by exploring the
vertical data format.
 First, we transform the horizontally formatted data into the vertical format by
scanning the data set once.
 The support count of an itemset is simply the length of the TID set of the itemset.
 Starting with k = 1, the frequent k-itemsets can be used to construct the
candidate (k + 1)-itemsets based on the Apriori property.
 The computation is done by intersection of the TID sets of the frequent k-
itemsets to compute the TID sets of the corresponding (k + 1)-itemsets.
 This process repeats, with k incremented by 1 each time, until no frequent
itemsets or candidate itemsets can be found.

 Besides taking advantage of the Apriori property in the generation of
candidate (k + 1)-itemset from frequent k-itemsets, another merit of this
method is that there is no need to scan the database to find the support of (k
+ 1)-itemsets (for k ≥ 1).
 This is because the TID set of each k-itemset carries the complete information
required for counting such support.
 However, the TID sets can be quite long, taking substantial memory space as
well as computation time for intersecting the long sets.
 To further reduce the cost of registering long TID sets, as well as the subsequent
costs of intersections, we can use a technique called diffset, which keeps
track of only the differences of the TID sets of a (k + 1)-itemset and a
corresponding k-itemset.

 For instance, in the example we have {I1} = {T100, T400, T500, T700, T800, T900}
and {I1, I2} = {T100, T400, T800, T900}.
 The diffset between the two is diffset({I1, I2}, {I1}) = {T500, T700}.
 Thus, rather than recording the four TIDs that make up the intersection of {I1}
and {I2}, we can instead use diffset to record just two TIDs, indicating the
difference between {I1} and {I1, I2}.
 Experiments show that in certain situations, such as when the data set contains
many dense and long patterns, this technique can substantially reduce the
total cost of vertical format mining of frequent itemsets.

 Which Patterns Are Interesting?—Pattern Evaluation Methods
 Most association rule mining algorithms employ a support–confidence
framework.
 Although minimum support and confidence thresholds help weed out or
exclude the exploration of a good number of uninteresting rules, many of the
rules generated are still not interesting to the users.
 Unfortunately, this is especially true when mining at low support thresholds or
mining for long patterns.
 This has been a major bottleneck for successful application of association rule
mining.

 Strong Rules Are Not Necessarily Interesting
 Whether or not a rule is interesting can be assessed either subjectively or
objectively.
 Ultimately, only the user can judge if a given rule is interesting, and this
judgment, being subjective, may differ from one user to another.
 However, objective interestingness measures, based on the statistics “behind”
the data, can be used as one step toward the goal of weeding out
uninteresting rules that would otherwise be presented to the user.
 “How can we tell which strong association rules are really interesting?” Let’s
examine the following example.

 Example -A misleading “strong” association rule.
 Suppose we are interested in analyzing transactions at AllElectronics with respect
to the purchase of computer games and videos.
 Let game refer to the transactions containing computer games, and video refer
to those containing videos.
 Of the 10,000 transactions analyzed, the data show that 6000 of the customer
transactions included computer games, while 7500 included videos, and 4000
included both computer games and videos.
 Suppose that a data mining program for discovering association rules is run on
the data, using a minimum support of, say, 30% and a minimum confidence of
60%.

 The following association rule is discovered:
buys(X, “computer games”) buys(X, “videos”)
⇒
(6.6)
[support = 40%, confidence = 66%].
 Rule (6.6) is a strong association rule and would therefore be reported, since its
support value of 4000 /10,000 = 40% and confidence value of 4000/ 6000 = 66%
satisfy the minimum support and minimum confidence thresholds, respectively.

 However, Rule (6.6) is misleading because the probability of purchasing videos is
75%, which is even larger than 66%.
 In fact, computer games and videos are negatively associated because the
purchase of one of these items actually decreases the likelihood of purchasing
the other.
 Without fully understanding this phenomenon, we could easily make unwise
business decisions based on Rule (6.6).
 This example also illustrates that the confidence of a rule A B can be
⇒
deceiving.
 It does not measure the real strength (or lack of strength) of the correlation and
implication between A and B.
 Hence, alternatives to the support–confidence framework can be useful in
mining interesting data relationships.

 From Association Analysis to Correlation Analysis
 As we have seen so far, the support and confidence measures are insufficient at
filtering out uninteresting association rules.
 To tackle this weakness, a correlation measure can be used to augment the
support–confidence framework for association rules.
 This leads to correlation rules of the form
A B [support, confidence, correlation]. (6.7)
⇒
 That is, a correlation rule is measured not only by its support and confidence but
also by the correlation between itemsets A and B.
 There are many different correlation measures available.
 Let us determine which would be good for mining large data sets.

 Lift is a simple correlation measure that is given as follows.
 The occurrence of itemset A is independent of the occurrence of itemset B if P(A B) =
∪
P(A)P(B); otherwise, itemsets A and B are dependent and correlated as events.
 This definition can easily be extended to more than two itemsets.
 The lift between the occurrence of A and B can be measured by computing
 If the resulting value of Eq. (6.8) is less than 1, then the occurrence of A is negatively
correlated with the occurrence of B, meaning that the occurrence of one likely leads to
the absence of the other one.
 If the resulting value is greater than 1, then A and B are positively correlated, meaning
that the occurrence of one implies the occurrence of the other.
 If the resulting value is equal to 1, then A and B are independent and there is no
correlation between them.

 Equation (6.8) is equivalent to P(B|A)/P(B), or conf(A B)/sup(B), which is also
⇒
referred to as the lift of the association (or correlation) rule A B.
⇒
 In other words, it assesses the degree to which the occurrence of one “lifts”
the occurrence of the other.
 For example, if A corresponds to the sale of computer games and B
corresponds to the sale of videos, then given the current market conditions,
the sale of games is said to increase or “lift” the likelihood of the sale of videos
by a factor of the value returned by Eq. (6.8).

 Example -Correlation analysis using lift.
 To help filter out misleading “strong” associations of the form A B from the data
⇒
of Previous example, we need to study how the two itemsets, A and B, are
correlated.
 Let game refer to the transactions of previous example that do not contain
computer games, and video refer to those that do not contain videos.
 The transactions can be summarized in a contingency table, as shown in Table
below.

 From the table, we can see that the probability of purchasing a computer
game is P({game}) = 0.60, the probability of purchasing a video is P({video}) =
0.75, and the probability of purchasing both is P({game, video}) = 0.40.
 By Eq. (6.8), the lift of Rule (6.6) is P({game, video})/(P({game}) × P({video})) =
0.40/(0.60 × 0.75) = 0.89.
 Because this value is less than 1, there is a negative correlation between the
occurrence of {game} and {video}.
 The numerator is the likelihood of a customer purchasing both, while the
denominator is what the likelihood would have been if the two purchases were
completely independent.
 Such a negative correlation cannot be identified by a support–confidence
framework.

 The second correlation measure that we study is the χ 2 measure.
 To compute the χ 2 value, we take the squared difference between the
observed and expected value for a slot (A and B pair) in the contingency table,
divided by the expected value.
 This amount is summed for all slots of the contingency table.
 Let’s perform a χ 2 analysis of Example 6.8.

`
 Example - Correlation analysis using χ 2 .
 To compute the correlation using χ 2 analysis for nominal data, we need the
observed value and expected value (displayed in parenthesis) for each slot of
the contingency table, as shown in Table below.

 From the table, we can compute the χ 2 value as follows:
 Because the χ 2 value is greater than 1, and the observed value of the slot (game,
video) = 4000, which is less than the expected value of 4500, buying game and
buying video are negatively correlated.
 This is consistent with the conclusion derived from the analysis of the lift measure in
previous example.

 A Comparison of Pattern Evaluation Measures
 The above discussion shows that instead of using the simple support–
confidence framework to evaluate frequent patterns, other measures, such as
lift and χ 2 , often disclose more intrinsic pattern relationships.
 How effective are these measures? Should we also consider other alternatives?
 Recently, several other pattern evaluation measures have attracted interest.
 Here, we present four such measures: all confidence, max confidence,
Kulczynski(Kulc) , and cosine.

 Given two itemsets, A and B, the all confidence measure of A and B is defined
as
 where max{sup(A), sup(B)} is the maximum support of the itemsets A and B.
 Thus, all conf(A,B) is also the minimum confidence of the two association rules
related to A and B, namely, “A B” and “B A.”
⇒ ⇒
 Given two itemsets, A and B, the max confidence measure of A and B is
defined as
 The max conf measure is the maximum confidence of the two association rules,
“A B” and “B A.”
⇒ ⇒

 Given two itemsets, A and B, the Kulczynski measure of A and B (abbreviated
as Kulc) is defined as
 It was proposed in 1927 by Polish mathematician S. Kulczynski.
 It can be viewed as an average of two confidence measures.
 Finally, given two itemsets, A and B, the cosine measure of A and B is defined
as

 Each of these four measures defined has the following property: Its value is only
influenced by the supports of A, B, and A B, or more exactly, by the
∪
conditional probabilities of P(A|B) and P(B|A), but not by the total number of
transactions.
 Another common property is that each measure ranges from 0 to 1, and the
higher the value, the closer the relationship between A and B.

 Example 6.10 Comparison of six pattern evaluation measures on typical data
sets.
 The relationships between the purchases of two items, milk and coffee, can be
examined by summarizing their purchase history in Table below, a 2 × 2
contingency table, where an entry such as mc represents the number of
transactions containing both milk and coffee.

 Table below shows a set of transactional data sets with their corresponding contingency
tables and the associated values for each of the six evaluation measures
 Let’s first examine the first four data sets, D1 through D4.
 From the table, we see that m and c are positively associated in D1 and D2, negatively
associated in D3, and neutral in D4.
 For D1 and D2, m and c are positively associated because mc (10,000) is considerably
greater than (1000) and (1000).
 Intuitively, for people who bought milk (m = 10,000 + 1000 = 11,000), it is very likely that they
also bought coffee (mc/m = 10/11 = 91%), and vice versa.

 The results of the four newly introduced measures show that m and c are strongly
positively associated in both data sets by producing a measure value of 0.91.
 However, lift and χ 2 generate dramatically different measure values for D1 and
D2 due to their sensitivity to
 In fact, in many real-world scenarios, is is usually huge and unstable.
 For example, in a market basket database, the total number of transactions
could fluctuate on a daily basis and overwhelmingly exceed the number of
transactions containing any particular itemset.
 Therefore, a good interestingness measure should not be affected by
transactions that do not contain the itemsets of interest; otherwise, it would
generate unstable results, as illustrated in D1 and D2.

 Similarly, in D3, the four new measures correctly show that m and c are strongly
negatively associated because the m to c ratio equals the mc to m ratio, that
is, 100/1100 = 9.1%.
 However, lift and χ 2 both contradict this in an incorrect way: Their values for
D2 are between those for D1 and D3.
 For data set D4, both lift and χ 2 indicate a highly positive association between
m and c, whereas the others indicate a “neutral” association because the
ratio of mc to equals the ratio of mc to , which is 1.
 This means that if a customer buys coffee (or milk), the probability that he or
she will also purchase milk (or coffee) is exactly 50%.

 “Why are lift and χ 2 so poor at distinguishing pattern association relationships in
the previous transactional data sets?”
 To answer this, we have to consider the null transactions.
 A null-transaction is a transaction that does not contain any of the itemsets being
examined.
 In our example, represents the number of null-transactions.
 Lift and χ 2 have difficulty distinguishing interesting pattern association
relationships because they are both strongly influenced by .
 Typically, the number of null transactions can outweigh the number of individual
purchases because, for example, many people may buy neither milk nor coffee.
 On the other hand, the other four measures are good indicators of interesting
pattern associations because their definitions remove the influence of (i.e.,
they are not influenced by the number of null-transactions).

 This discussion shows that it is highly desirable to have a measure that has a
value that is independent of the number of null-transactions.
 A measure is null-invariant if its value is free from the influence of null-
transactions.
 Null-invariance is an important property for measuring association patterns in
large transaction databases.
 Among the six discussed measures, only lift and χ 2 are not null-invariant
measures.
 “Among the all confidence, max confidence, Kulczynski, and cosine measures,
which is best at indicating interesting pattern relationships?”

 To answer this question, we introduce the imbalance ratio (IR), which assesses the
imbalance of two itemsets, A and B, in rule implications.
 It is defined as
 where the numerator is the absolute value of the difference between the support
of the itemsets A and B, and the denominator is the number of transactions
containing A or B.
 If the two directional implications between A and B are the same, then IR(A,B) will
be zero.
 Otherwise, the larger the difference between the two, the larger the imbalance
ratio.
 This ratio is independent of the number of null-transactions and independent of the
total number of transactions.

 Example -Comparing null-invariant measures in pattern evaluation.
 Let’s examine data sets D5 and D6, shown in Table below, where the two
events m and c have unbalanced conditional probabilities.

 That is, the ratio of mc to c is greater than 0.9.
 This means that knowing that c occurs should strongly suggest that m occurs also.
 The ratio of mc to m is less than 0.1, indicating that m implies that c is quite unlikely
to occur.
 The all confidence and cosine measures view both cases as negatively associated
and the Kulc measure views both as neutral.
 The max confidence measure claims strong positive associations for these cases.
 The measures give very diverse results!
 “Which measure intuitively reflects the true relationship between the purchase of
milk and coffee?”
 Due to the “balanced” skewness of the data, it is difficult to argue whether the two
data sets have positive or negative association.

 From one point of view, only mc/(mc + ) = 1000/(1000 + 10,000) = 9.09% of
milk-related transactions contain coffee in D5 and this percentage is 1000/(1000
+ 100,000) = 0.99% in D6, both indicating a negative association.
 On the other hand, 90.9% of transactions in D5 (i.e., mc/(mc + ) =
1000/(1000 + 100)) and 9% in D6 (i.e., 1000/(1000 + 10)) containing coffee
contain milk as well, which indicates a positive association between milk and
coffee.
 These draw very different conclusions.
 For such “balanced” skewness, it could be fair to treat it as neutral, as Kulc does,
and in the meantime indicate its skewness using the imbalance ratio (IR).
 According to Eq. (6.13), for D4 we have IR(m,c) = 0, a perfectly balanced case;
for D5, IR(m,c) = 0.89, a rather imbalanced case; whereas for D6, IR(m,c) = 0.99,
a very skewed case.
 Therefore, the two measures, Kulc and IR, work together, presenting a clear
picture for all three data sets, D4 through D6.

 In summary, the use of only support and confidence measures to mine
associations may generate a large number of rules, many of which can be
uninteresting to users.
 Instead, we can augment the support–confidence framework with a pattern
interestingness measure, which helps focus the mining toward rules with strong
pattern relationships.
 The added measure substantially reduces the number of rules generated and
leads to the discovery of more meaningful rules.

1.pptx .

More Related Content

Similar to 1.pptx .

More from SaiM947604

Recently uploaded

1.pptx .