Please enable JavaScript.
Coggle requires JavaScript to display documents.
Larsen Chapters 17 and 18 (Chapter 17: Evaluate Model Performance (At this…
Larsen Chapters 17 and 18
Chapter 17: Evaluate Model Performance
At this point we have a very limited understanding of how each algorithm works. As AutoML users we must commit to either knowing all algorithms at a technical level or knowing the function of these algorithms conceptually
Think of it like going from riding a bike > riding an electric bike > riding a moped > riding a motorcycle
Presenting to Management
Should never be about explaining how the algorithm works, rather be about their performance characteristics
Understanding Models
There is value in having a general understanding of how algorithms work so as to be more effective in creation and validation
ROC Curve Screen
Understanding Measures
Positive Predictive Value (Precision)
TP/(TP + FP) - what portion of the cases the mode considers positives are actually positives
True Positive Rate (Sensitivity)
TP / (TP + FN) - What proportion of the positive cases is the model capable of finding
Accuracy
TP + TN / All Cases - What proportion of model predictions are correct
Chapter 18: Comparing Model Pairs
DataRobot has great tools to help compare models
Prioritizing Modeling Criteria and Selecting a Model
There are 5 criteria to consider
Speed to Build Model
Familiarity With Model
Prediction Speed
Insights
Predictive Accuracy
Often, prioritizing predictive accuracy comes at the expense of the other four criteria and as a data scientist it is important to find the optimal combination