Chapter 19. Interpret Model (19.1 Feature Impacts on Target (19.2 The…
Chapter 19. Interpret Model
19.1 Feature Impacts on Target
19.2 The overall impact of features on the Target without consideration of other features.
The overall impact of a feature without consideration for the impact of other features treats each feature as a standalone effect on the target.
The importance score (green line) is exceedingly useful because it allows a data scientist to focus attention on the features most likely to yield additional predictive value if misinterpreted by the AutoML, such as through misinterpretation of the variable
These scores are not fully reliable indicators of the value of a feature
In short, while they provide a useful way to sort features, importance scores should not be relied on for feature selection and model interpretation.
19.3 The Overall Impact of a Feature Adjusted for the Impact of other Features.
Feature Impact, will calculate each value of each feature in
the context of this model. DataRobot does this by randomly shuffling the values of one feature within the validation data segment (thereby removing its ability to have a meaningful impact on the target). DataRobot then examines the model’s performance relative to the model that retained all the features. The extent to which the model with the randomly shuffled feature does worse than the original model is assigned as the feature’s value. This procedure is done with every feature (all the features not being examined retain their original data). Once each feature has been scored this way, the most important feature is scaled to a score of 100%, and the other features are scaled relative to it.
DataRobot has conveniently placed an option to select a set of top features as a new feature list. All that is necessary is to give the new list a name and select how many of the top features will populate it. A new model run can then be done using this feature list. Creating models with fewer features is generally a good idea to avoid overfitting and can also reduce problems due to changes in the databases and sources of data.
19.4 The Directional Impact of Features on Target
The third type of relationship is what has been termed in this book the directional impact of the feature.
Go now to the Insights screen Once there, select Variable Effects which will open the view displayed in Figure 19.3. What is shown is the result of a logistic regression analysis.
For many datasets, logistic regression does well enough that such new runs are not necessary, but, as it is not always the case, knowing how to rerun a regression model at a greater data quantity may prove valuable in future projects.
A technical note on directional impacts: as you get more comfortable with DataRobot as a user, consider looking at the leaderboard for any directional impact information. Any model with a Coefficients tab (See Figure 19.6), such as this
Elastic-Net Classifier, would likely be a better source of such coefficients if it is ranked higher in the leaderboard than the logistic regression model that appears in the insights tab. (After all, these kinds of insights are most valuable in cases where
the models are as predictive as possible.)
19.5 The Partial Impact of Features on Target
The Model X-Ray constructs a list of features ranked by
their influence on the target as denoted by the size of the green line under each feature name in the left pane. This information is the same as that used to rank features under the Data tab,
19.6 The Power of Language
AutoML makes analytics easy enough that, for the first time, it is easier to train subject matter experts in machine learning than it is to train a machine learning expert in the subject matter.
A word cloud represents the words that have the highest
coefficients (remember adding these up to get an idea of the likelihood of a case being readmitted?). The intensity of the red or blue colors indicates the size of their coefficient. By hovering over a term (one or more words in order from the text
feature), the coefficient of that specific term is shown.
The key to interpreting the word cloud,
is to simply analyze large terms with vivid colors.
1 more item...
Hotspots visualization uses the RuleFit Classifier, which is an immediate red flag, considering that this is one of the
algorithms that was stalled at 16% of the data during model creation.
The Hotspot screen shows the most relevant (up to four) combinations of features and their effect on the target.60 Think of this diagram as a set of Venn diagrams where the largest and most overlapping hotspots are organized in the middle. Just
as for the word cloud, the deeper the tone of the blue and the red, the more of an impact that particular combination of features
1 more item...
19.8 Reason Codes
The left and right thresholds allow for the specification of the probability cutoffs to be used in this view. For example, the left (blue) threshold states here that reason codes are desired for all patients with probabilities of readmission ranging between
0.0 and 0.22. The rightmost (red) threshold specifies that reason codes are also desired for any patient with a probability between 0.585 and 1.0.
1 more item...
In DataRobot, the four types of relationships a feature can have with a target are as follows:
The overall impact of a feature without consideration of the impact of other
The overall impact of a feature adjusted for the impact of other features.
The directional impact of a feature.
The partial impact of a feature