Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapter 7: Decision Analytic Thinking 1: What is a Good Model (Evaluating…
Chapter 7: Decision Analytic Thinking 1: What is a Good Model
Evaluating Classifiers
Negative
benign
Positive
Cause for alarm
Classifier Accuracy
Any general measure of classifier performance
accuracy
proportion of correctness
Confusion Matrix
Class Label and Class Predicted by classifier
separates decisions made by cassifier
Errors of the classifiers are false positives
Problems with Unbalanced Classes
As class distribution becomes more skewed
evaluation based on accuracy goes down
One class more prevalent than another
accuracy becomes misleading
Accuracy is the wrong thing to measure
Problems with unequal costs and benefits
No distinction between false positive and false negative
once aggregated produce expected profit
Generalizing beyond Classification
Grouping and predicting based off similarity
Expected Value
Decomposes data analytic thinking into
structure of problem
elements of analysis extracted from data
elements that need to be acquired from another source
Framing classifier use
expected values calculation expresses using the model
Framing classifier evaluation
Comparing models
In aggregate how well does each model do
Error Rates
error vs correct decisions
Cost and Benefits
Costs and benefits often not estimated from data
Depends on external info
Profitability
Evaluation, Baseline performance, and Implications for Investments in Data
Baseline
Compared against
majority classifier
Conditional model
predicts differently based on value of features