Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapter 17-18 (Chapter 17: Evaluate Model Performance (Receiver Operating…
Chapter 17-18
Chapter 17: Evaluate Model Performance
Know conceptually how the models work, as you never know which model will turn out the best
Understand the leaderboard metric
FVE Binomial provides a sense of how much of the variance in the dataset has been explained
equivalent to an R2-value
Knowing and using all metrics available will improve model selection
tree-based algorithms build on the logic of the decision-tree classifier, which is to repeatedly find the most predictive feature at that instance and split it into two groups that are as homogenous as possible
Receiver Operating Characteristics (ROC) Curve
each case is assigned a color (green or purple depending on its true value). After its color is assigned, the specific case falls atop the existing cases at their respective assigned probabilities, building the “mountain” for that color.
the less overlap the better
Threshold: probability where the decision changes from false to true, in a binomial prediction case
Confusion matrix shows real vs predicted readmittance
The type of mistake a model makes matters, running a red vs stopping on a green
Chapter 18: Comparing Model Pairs
Compare how each model does at predicting readmits in different bins along the data
Difference seen in the model comparison screen may be undetectable in other screens, such as the ROC Curve
Small discrepancies may make a very large (or small) real world difference
Dual Lift shows where models disagree, and which is better in those senarios
5 Cirteria
Predictive accuracy.
Prediction speed.
Speed to build model.
Familiarity with model.
Insights.