Decision Analytic Thinking I: What Is a Good Model? (A key analytical…
Decision Analytic Thinking I: What Is a Good Model?
One worthy of attention, an alarm
Uninteresting or benign
Easy to measure
Measure of classifier performance
Accuracy = # of correct decisions made / total # of decisions made
= 1 - error rate
Way to decompose & count different types of correct & incorrect decisions made by a classifier
Use confusion matrix
N classes is an n x n matrix
Columns actual class label
Rows predicted classes
Separates decisions made by classifier
True classes P(ositive) or N(egative)
Predicted classes Y(es) or N(o)
Errors of classifiers are false positives & false negatives
Class distribution becomes more skewed, accuracy breaks down
Even when skewed not great, domain where one class more prevalent than another accuracy may be misleading
Bottom-line is accuracy simply is the wrong thing to measure
Both models classify 80% of balanced population
1 more item...
Problems with unequal costs & benefits
Makes no distinction between false positive/negative errors
Believes are the same, not the case
False positive error - wrongly informed patient of cancer
Opposite, has cancer but is wrongly told they do not - false negative
Estimate cost or benefit of each decision a classifier makes
Once aggregated produces expected profit
Generalizing beyond classification
Return to questions ..
What is important in the application?
What is the goal?
Are we assessing the results of data mining appropriately given the actual goal?
A key analytical framework: Expected Value
Decomposes data-analytic thinking to ..
Structure of the problem
Elements of the analysis that can be extracted from the data
Elements of the analysis that need to be acquired from other sources
Outcomes are enumerated
Interested in maximizing profit
EV = p(o1)
v(o1) + p(o2)
v(o2) + p(o3) * v(o3) ....
O1/2/3 are possible decision outcome
p(01) is probability
v(o1) is its value
Targeted Marketing Example
Use to frame classifier evaluation
Want to shift focus from individual decisions to collections
Neccessary to compare one model to another
How well does each model make decisions in aggregate, expected value
Expected profit calculation in aggregate model
Estimated from tallies in confusion matrix
Cost and benefits
Correct classifications - true positives & negatives correspond with benefits b(Y, n) and b(N,n)
Incorrect classifications - false negatives & positives correspond with benefit and costs c(Y,n) & c(N,p)
Can be summarized in a 2x2 matrix
Expected profit equation (too long to type, check book)
Positive example very rare, contribution to expected profit very small
Evaluation, Baseline Performance, and Implications for Investments in Data
Data informative, but important to consider reasonable baseline which to compare model performance
Important to know for data science team so they can track if performance is improving
Beating random model fairly easy
One good baseline is the majority classifier
naive classifier that always chooses the majority class of the training dataset
Complex model that considers small amount of feature info
One variable could lead to regression or classification of that one variable
Ex: Tree induction to build "decision stump"