Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapter 17: Evaluate Model Performance (ROC Curve (Ex: Cross validation,…
Chapter 17: Evaluate Model Performance
Introduction
AutoML should improve a user's understanding of the problem by providing visualization of the interactions between the features and target
Important to know algorithms at a technical level or knowing their functions conceptually
Can explain performance characteristics rather than how the algorithm works
Acquire the skill of understanding a model's performance and a model's business context
While LogLoss optimizes overall success, it does not constitute a reasonable measure of how far from the target predictions are at the level of the average case
Fraction of Variance Explained (FVE) Binomial
provides a sense of how much of the variance in the dataset has been explained and is equivalent to an R^2 value
A Sample Algorithm and Model
Decision Tree Classifier: repeatedly finds the most predictive feature at that instance and split it into two groups that are as internally homogeneous as possible
Can discover the origin of algorithms and the parameters used
3 steps of a decision tree classifier
Find the most predictive feature (the one that best explains the target) and place it at he root of the tree
Split the feature into two groups at the point of the feature where the two groups are as homogenous as possible
Repeat step 3 for each new branch
IF-THEN statements
Best models will be examined in their performance as applied to previously unseen cross validation, noting model predictions vs. true readmission data for each data point
ROC Curve
Different aspect of quality
Receiver Operating Characteristics Curve: where several measures of model success exist beyond the original optimization metic, LogLoss
Ex: Cross validation, assigned probabilities, density distribution
Cross validation and validation scores should not differ wildly
Table 17.1 Measures & their meaning that can be calculated based on the confusion matrix
Confusion Matrix- predicted vs. actual
Based on the 4 quadrants of the confusion matrix, the main dynamic measure commonly used is the Area Under the Curve of the ROC Curve
A good AUC score is having a low FPR and a high TPR at any threshold
Evaluate the model performance at several prediction distribution thresholds
Using the Lift Chart for Business Decisions
Lift chart: sorts all validation cases by their probability
Having blue and orange lines that are fully overlapping indicate a strong model
Ex: Can construct campaigns based on results and calculate how much it costs per reach; profit charts
Enable Down Drill: to download actual predictions in order to better understand what is going on inside the model