Ch.7 What is a Good Model? (Using expected Value to Frame Classifier…
Ch.7 What is a Good Model?
Bad positives and Harmless negatives
The Confusion Matrix
2by2 table. horisontal prédiction, vertical outcome.
Plain Accuracy and Its Problems
accuracy = Number of correct decisions made/total number of decisions. Also 1-error rate.
Problems with Unbalanced Classes
Not representative of reality. You might have high rate just because the probability of one is greater.
Problems with Unequal Cost and Benefits
No distinction between false pos and false neg. Very different types of error. Churn example false pos. give incentive but for no use vs false neg leaving without being caught by the model.
Generalizing Beyond Classification
Goal? and importance
A Key Analytical Framework:Expected value
Expected value: EV= p(o1)
v(o2)+... each oX is a possible decision outcome, p it probability and v its value. The prob can be found in data value is more objective
Using expected Value to Frame Classifier Use
Expected benefit of targeting = PR(x)
Using expected Value to Frame Classifier Evaluation
Cost and benefits
True x = benefit False x=cost
Evaluation, Baseline Performance , and Implications for Investments in Data
Expected value, Confusion matrix and its costs and benefits