Please enable JavaScript.
Coggle requires JavaScript to display documents.
Visualizing Model Performance: Chapter 8 (Profit Curves (Visualizations in…
Visualizing Model Performance: Chapter 8
Ranking instead of Classifying
Rank set of cases by scores, then take actions on cases at top of ranked lists
Top "n" cases
"Budget" for actions
In some instances, classifier decisions should be very conservative
"Confusion Matrix"
Comparing different rankings?
Choosing a proper threshold
Profit Curves
Visualizations in the form of curves
Each curve based on idea of examining effect of threshold
As we move threshold "down" the ranking, we get additional instances predicted as being positive rather than negative
Expected value
List of instances and their predicted scores
Consumers ordered from highest to lowest probability of accepting an offer base on some model
Shows profit can go negative-- not always though
Costs and the calss ratio
"Going into the red"
"Budget" - fixed amount of money
ROC Graphs and Curves
Class Priors
proportion of positive and negative instances in the target population
Base Rate
Costs and benefits
Expected profit is specifically sensitive to these relative levels
If both class priors and cost-benefit estimates are known and expected to be stable, profit curve are a good choice for visualizing performance
In many domains these conditions are uncertain or unstable
Receiver Operating Characteristics
Labeled A through E
Different classifiers
Discrete
Tp rate
Fp rate
Cumulative Response and Lift Curves
Need to communicate results to key stakehodlers
More intuitive
Cumulative Response Curve
Also called a "Lift curve"
Closely related to ROC curve
Percentage of positives correctly classified
Diagonal line x=y represents random performance
If you target 20% of all instances completely random, you should target 20% of positives as well.
Lift is the degree to which it "pushes up" the positive instances in a list above the negative instances
Giving a "two times" lift means that at the chosen threshold, the lift curve shows that the model's targeting is twice as good as random