Visualizing Model Performance
importance
high level detail
prove specific point more quickly/strongly
profit margin
methods
ranking
cost/benefits
classify the objects
issues
confusion matrix
classifier + threshold
as threshold is lowered, instances move from N to Y (row)
how to choose best threshold?
how to compare diff. rankings?
where we desire it to be
Profit above 0
Profit Curve
different classifiers, different profit curves
can always go negative!
that's biz
budgetary restraints
changes best classifer
changes point of operation
critical conditions
class priors: proportion of positive and negative instances in the target population
base rate
costs and benefits: expec. profit sensitive to values defined within the classifier
if both known, then good decision for model
often not
Receiver Operating Characteristics (ROC) Curve
false positive rate x axis (costs), true positives rate y axis (benefits)
depicts tradeoffs
discrete classifier: outputs only class label (no ranking)
produces an fp and tp rate pair
conservative results: to the left, toward x axis
independent of class proportions and actual costs and benefits
Area Under the ROC Curve (AUC)
ranges from 0-1
useful to summarize performance with single number
click to edit
useful when you don't know operating conditions
Cumulative Response Curve
lift curve
plots hit rate
percentage of postives correctly classified as function of population that is targeted
target increasingly larger proportions
how does it respond to increase in model sample?
Performance
train data
data used to produce model
test data
cross fold validation
Comparing results of different models
class tree
log. regression
k nearest neighbor
naive bayes
different points of sample data will produce a better perfomring model
break complex data problems into business tasks
compare results across metrics (AUC/ROC/etc)
how many separate prediction clusters will my train data use?
data used to test model
CEOs probably wont understand ROC curves