Please enable JavaScript.
Coggle requires JavaScript to display documents.
Visualizing Model Performance (methods (Receiver Operating Characteristics…
Visualizing Model Performance
importance
high level detail
prove specific point more quickly/strongly
profit margin
CEOs probably wont understand ROC curves
break complex data problems into business tasks
methods
ranking
cost/benefits
classify the objects
issues
confusion matrix
classifier + threshold
as threshold is lowered, instances move from N to Y (row)
how to choose best threshold?
where we desire it to be
Profit above 0
how to compare diff. rankings?
Profit Curve
different classifiers, different profit curves
budgetary restraints
changes best classifer
changes point of operation
can always go negative!
that's biz
critical conditions
class priors: proportion of positive and negative instances in the target population
base rate
costs and benefits: expec. profit sensitive to values defined within the classifier
if both known, then good decision for model
often not
Receiver Operating Characteristics (ROC) Curve
false positive rate x axis (costs), true positives rate y axis (benefits)
depicts tradeoffs
discrete classifier: outputs only class label (no ranking)
produces an fp and tp rate pair
conservative results: to the left, toward x axis
independent of class proportions and actual costs and benefits
Area Under the ROC Curve (AUC)
ranges from 0-1
useful to summarize performance with single number
useful when you don't know operating conditions
Cumulative Response Curve
lift curve
plots hit rate
percentage of postives correctly classified as function of population that is targeted
target increasingly larger proportions
how does it respond to increase in model sample?
Performance
train data
data used to produce model
test data
data used to test model
cross fold validation
how many separate prediction clusters will my train data use?
Comparing results of different models
class tree
log. regression
k nearest neighbor
naive bayes
different points of sample data will produce a better perfomring model
compare results across metrics (AUC/ROC/etc)