Data Science In Pune

data science in pune

244 views

Embed
Email

From

Username or Email (please add comma after each username or email)

Name	Email

Back

Menu 3

Eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo.

Ravali470d

Uploaded on Oct 10, 2018

Category Education

Data for a Data Scientist is what Oxygen is to Human Beings. This is also a profession where statistical adroit works on data

Category Education

Comments

data science in pune

Data Mining

Market
Basket

Analysis

Affinity
Analysis

Relationship
Mining

Association Rules

Market Basket Analysis

• Large number of transaction records through data collected using bar-code scanners

• Each record = All items purchased on a single purchase transaction

Association Rules

• What item goes with what

• Are certain groups of items consistently purchased together

• What business strategies will you device with this knowledge

Association Rules

• Products shelf placement – a specific product beside another

• Selling of prominent shelves – Slotting Fees

• Stocking – Supply Chain Management

• Price Bundling – Combo offers. How?

Source: http://www.economist.com/news/business/21654601-supplier-rebates-are-heart-some-supermarket-chains-woes-buying-up-shelves

https://en.wikipedia.org/wiki/Association_rule_learning

Association Rules – Cell phone faceplates

A store sells accessories for cellular
phones runs a promotion on
faceplates

OFFER!
Buy multiple faceplates from a choice of
6 different colors & get discount

How would you help store managers
device strategy to become more
profitable

Association Rules – Cell phone faceplates

Transaction # Faceplate colors purchased Transaction # Red White Blue Orange Green Yellow

1 Red White Green 1 1 1 0 0 1 0

2 White Orange 2 0 1 0 1 0 0

3 White Blue 3 0 1 1 0 0 0

4 Red White Orange 4 1 1 0 1 0 0

5 Red Blue 5 1 0 1 0 0 0

6 White Blue 6 0 1 1 0 0 0

7 White Orange 7 0 1 0 1 0 0

8 Red White Blue Green 8 1 1 1 0 1 0

9 Red White Blue 9 1 1 1 0 0 0

10 Yellow 10 0 0 0 0 0 1

List Format Binary Matrix Format

Association Rules are probabilistic “if-then” statements

2 Main Ideas:
 Examine all possible “if-then” rule formats
 Select rules, which indicates true dependence

Association Rules – Cell phone faceplates

Rules for { Red, White, Green}

1. If {Red, White} then {Green}

2. If {Red, Green} then {White}

3. If {White, Green} then {Red}

4. If {Red} then {White, Green}

5. If {White} then {Red, Green}

6. If {Green} then {Red, White}

Problem

• Many rules are possible

• How to select the
TRUE/GOOD rules from
all generated rules?

Association Rules – Terminology

• If {Red, White} then {Green}

• If Red & White phone faceplates are purchased, then Green
faceplate is purchased

 Antecedent: Red & White
 Consequent: Green

“IF” part = Antecedent = A

“THEN” part = Consequent = C

Association Rules – Performance Measures

Support

1
Confidence Lift

2 3

Association Rules – Support

Support

• Consider only combinations that occur with
higher frequency in the database

• Support is the criterion based on frequency

Percentage / Number of transactions in which
IF/Antecedent & THEN / Consequent appear in
the data

Mathematically:
# transactions in which A & C appear together
_____________________________________

Total no. of transactions

Support - Calculation

• What is the support for
“if White then Blue”?

1. 4
2. 40%
3. 2
4. 90%

Transaction # Faceplate colors purchased

1 Red White Green

2 White Orange

3 White Blue

4 Red White Orange

5 Red Blue

6 White Blue

7 White Orange

8 Red White Blue Green

9 Red White Blue

10 Yellow

• What is the support for
“if Blue then White”?

1. 4
2. 40%
3. 2
4. 90%

Support - Problem

• Generating all possible rules is exponential in the
number of distinct items

• Solution:
Frequent item sets using Apriori Algorithm

Apriori Algorithm For k products:

Set minimum support criteria

Generate list of one-item sets that meet the support
criterion

Use list of one-item sets to generate list of two-item sets
that meet support criterion

Use list of two-item sets to generate list of three-item sets
that meet support criterion

Continue up through k-item sets

Support – Criterion = 2

Transaction # Faceplate colors purchased

1 Red White Green

2 White Orange

3 White Blue

4 Red White Orange

5 Red Blue

6 White Blue

7 White Orange

8 Red White Blue Green

9 Red White Blue

10 Yellow

Item set Support (Count)

{Red} 5

{White} 8

{Blue} 5

{Orange} 3

{Green} 2

{Red, White} 4

{Red, Blue} 3

{Red, Green} 2

{White, Blue} 4

{White, Orange} 3

{White, Green} 2

{Red, White, Blue} 2

{Red, White, Green} 2

Create rules from
frequent item sets only

Support Criterion Example

Rules for { Red, White, Green}

1. If {Red, White} then {Green}

2. If {Red, Green} then {White}

3. If {White, Green} then {Red}

4. If {Red} then {White, Green}

5. If {White} then {Red, Green}

6. If {Green} then {Red, White}

How good are these
rules beyond the point
that they have high
support?

Association Rules – Confidence

Confidence

• Percentage of If/Antecedent transactions that
also have the Then/Consequent item set

Mathematically:

P (Consequent | Antecedent) = P(C & A) / P(A)

# transactions in which A & C appear together
_____________________________________

# transactions with A

Confidence - Calculation

• What is the confidence
for “if White then Blue”?

1. 4/5
2. 5/8
3. 5/4
4. 4/8

Transaction # Faceplate colors purchased

1 Red White Green

2 White Orange

3 White Blue

4 Red White Orange

5 Red Blue

6 White Blue

7 White Orange

8 Red White Blue Green

9 Red White Blue

10 Yellow

• What is the confidence
for “if Blue then White”?

1. 4/5
2. 5/8
3. 5/4
4. 4/8

Confidence - Weakness

• If antecedent and consequent have:

High Support => High / Biased Confidence

Association Rules – Lift Ratio

Lift Ratio

Confidence / Benchmark
confidence

Benchmark assumes independence between
antecedent & consequent:

P(antecedent & consequent) = P(antecedent) X P(consequent)Benchmark confidence
P(C|A) = P(C & A) / P(A) = P(C) X P(A) /P(A) = P(C)

# transactions with consequent item sets
_____________________________________

# transactions in database

Interpreting Lift

• Lift > 1 indicates a rule that is useful in finding consequent item
sets

• The rule above is much better than selecting random transactions

Lift - Calculation

• What is the Lift for “if White then Blue”?
1. 4/8
2. 5/10
3. 4/5
4. 1

Transaction # Faceplate colors purchased

1 Red White Green

2 White Orange

3 White Blue

4 Red White Orange

5 Red Blue

6 White Blue

7 White Orange

8 Red White Blue Green

9 Red White Blue

10 Yellow

Rules selection process

Generate all rules that meet
specified Support & Confidence

 Find frequent item sets based on
Support specified by applying
minimum support cutoff

 From these item sets, generate rules
with defined Confidence. By filtering
remaining rules select only those with
high Confidence

Rules
Inputs Data

# Transactions in Input Data 10

# Columns in Input Data 6

# Items in Input Data 6

# Association Rules 8

Minimum Support 2

Minimum Confidence 70.00%

List of
Rules

Rule: If all Antecedent items are purchased, then with Confidence percentage Consequent items will also be
purchased.

Row ID Confidence %
Antecedent

(A)
Consequent

A
Support for

C
Support for

A & C
Lift

Ratio

8 100 green red & white 2 4 2 2.5

4 100 green red 2 5 2 2

6 100 white & green red 2 5 2 2

3 100 orange white 3 8 3 1.25

5 100 green white 2 8 2 1.25

7 100 red & green white 2 8 2 1.25

1 80 red white 5 8 4 1

2 80 blue white 5 8 4 1

Alarming!

 Random data can generate apparently
interesting association rules

 More the rules you produce, greater the
danger

 Rules based on large numbers of records
are less subject to this danger

Profusion of rules

Applications
• What if Product & Stores are selected as a tuple for analysis?

• What if crimes in different
geographies for each week
is known?

Narcotics

Robbery

AssaultBattery Narcotics
Public
Peace

Violation

Recap with an example
• How can you use the information if you know about the

purchase history of customers in a specific geography?

• Supermarket database has 100,000 POS transactions

• 2000 transactions include both Strepsils & Orange Juice

• 800 of the above 2000 include Soup purchases

Recap with an example
• What is the support for rule “IF (Orange Juice & Strepsils) are purchased

THEN (Soup) is purchased on the same trip”?

1. 0.8 %
2. 2 %
3. 40 %

• What is the confidence for rule “IF (Orange Juice & Strepsils) are purchased
THEN (Soup) is purchased on the same trip”?

1. 0.8 %
2. 2 %
3. 40 %

Recap with an example

• What is the lift ratio for rule “IF (Orange Juice & Strepsils) are purchased
THEN (Soup) is purchased on the same trip”?

Sequential Pattern Mining

Purchases / events occur at

the same time

• If person X has taken “Data
Mining Unsupervised” training
in 1st Quarter, Person X has

also taken “Data Mining
Supervised” training in 2nd

Quarter

• Based on the statement
above, recommend “Data
Mining Supervised” training to
those who have enrolled for

“Data Mining Unsupervised”

NOT

IT IS

Association Rules vs. Sequential Pattern Mining

• Look for temporal patterns

• Order/sequence of a & b matters for a rule “b follows a”

• However, what happens in between a & b doesn’t matter

• In phone faceplates dataset:
 Association among items, which were bought within the

same week were discovered

 How about finding what they would buy next week or the
week after, if they had bought ‘x’ in this week?

Applications

• Identify the appropriate Basket

• Identify popular taxi routes

 Sequential pattern from GPS tracks;
spatiotemporal records of taxi
trajectories

 First cluster collocated customers

THANK YOU