Chapter 18: Comparing Model Pairs (Model Comparison (To understand the…
Chapter 18: Comparing Model Pairs
Selecting a Model
At this point, one of the models produced should stand out as a winner. Usually, there is a high demand for rationale if selecting a model that is not at the top of the leaderboard
To understand the difference between the overall best model (the ENET Blender, M101) and the best non-blender model (XGBoost model M63), DataRobot provides a way to examine the two against each other.
In the upper left of the screen, select
Model Comparison: allows the selection of two models from the leaderboard in addition to
auto-selecting the top model, placed in the far left position.
Click Change Model on the right-most model and search for M63. Select this model and click on the chart if necessary. The left model (the ENET Blender) is shown in a tan color, whereas the right model (the XGBoost model) is shown in blue. Select Lift, and make sure that Lift Data Source is set to Cross Validation, and number of bins is set to 15.
The subsequent chart orders each algorithm’s predictions by probability from high to low and splits each into 15 bins, calculating the average number of readmits (values of one) in each bin and ordering those bins from right to left on the X-axis. The Y-axis displays the accuracy of each bin.
Prioritizing Modeling Criteria
When deciding which model to select, there are five criteria to consider.
which measure DataRobot should use for its
hyperparameter optimization, leaderboard ranking, and model selection
How long can the hospital staff afford to wait for that process to reach a conclusion?
Speed to build model.
reflects how long it takes to train a model.
Familiarity with model.
assumes that the data scientist (you) is an expert on one of the algorithms and are able to understand the exact meaning of its results
based on an assumption that different algorithms make different statistics available.
Important pages to note
Prediction speed is not an issue in this specific instance of the hospital setting, the model will not need to be rebuilt often, and the current speed is acceptable, even for the ENET blender. Therefore these measure can be considered secondary in model selection for this case
This decision can also be justified due to the fact that the simpler models that are more familiar, such as the logistic regression and decision tree models, did not do well enough to be realistically considered