Please enable JavaScript.
Coggle requires JavaScript to display documents.
Ch.17-18 (introduction (For example, if a model were construct that has no…
Ch.17-18
introduction
In datarobot, there is no way to know a priori which algorithm will be elevated to the top of the leaderboard.
-
While LogLoss does a great job of optimizing overall success, for most users, it does not constitute a reasonable measure of how far from the target predictions are at the level of the average case
For example, if a model were construct that has no better ability to predict a patient outcome (readmission) than
random assignment (heads or tails on a coin), that model would have a cross validation LogLoss score of 0.672.
There are measures, however, that make such
evaluations in a more understandable manner, such as Fraction of Variance Explained (FVE) Binomial (Figure 17.2).
FVE Binomial provides a sense of how much of the variance in the dataset has been explained and is equivalent to an R^2-value
in the simpler terms, this metric states how far off, percent-wise, the model is from fully explaining who will be readmitted
-
ROC curve
The Receiver Operating Characteristics (ROC) Curve screen, so named because of the ROC curve in the bottom left corner, is where several central measures of model success exist beyond the original optimization metric, LogLoss.
True Positive Rate (TPR), TPR measures what proportion of positive cases (readmissions) have been correctly identified and is calculated by TP/(TP+FN) or the number of cases in the bottom right quadrant divided by the number of cases in the two bottom quadrants
positive predictive value (PPV). This measure is derived from the two rightmost quadrants and is more often called precision. It is calculated as TP?(TP+FP), or the number of cases in the bottom right divided by the number of cases in the right two quadrants.
-
-
model comparison
To save processing time, the model selected was the best non-blender model due to the fact that blender models are generally beyond an easy explanation and take longer for model X-Ray and Feature Impacts to calculate.
To understand the difference between the overall best model (the ENET Blender, M101) and the best non-blender model (XGBoost model M63), DataRobot provides a way to examine the two against each other.
Model comparison: This panel allows the selection of two models from the leaderboard in addition to auto- selecting the top model, placed in the far left position.
-