Chapter 20: Communicate Model Insights (There are six types of information…
Chapter 20: Communicate Model Insights
There are six types of information that should be communicated during a presentation:
Model quality metrics (confusion matrix)
Areas where a model struggles (potential for improvement through more
data––features & cases)
Most predictive features for model building
Feature types especially interesting to management (e.g., insights into the
business problem and unknowns uncovered during the modeling process)
Recommended business actions (i.e., to implement model or not, any
business decisions to implement at various probability thresholds, and how will doing so change practice?)
20.1 Unlocking Holdout
it is important to check that there haven’t been any mistakes made in the model creation process, this is done by releasing the holdout data
20.2 Business Problem First
We will create a machine-learning model capable of detecting which patients are likely to be readmitted within 30 days of discharge and develop educational and support programs targeting these patients.
60% of our patients are discharged to home without home health services; this group mirrors the overall patient population in terms of average readmission.
We are especially interested in detecting cases where these patients are highly likely to be readmitted.
20.3 Pre-processing and Model Quality Metrics
Because it is now time to put your career on the line with model M101, the holdout sample just released will be used. Though this model performs slightly worse on the holdout data, in all likelihood, this result is closer to realistic model performance.... it is now time to plan the model quality metrics part of the presentation.
First, it is necessary to extract the predicted probabilities for all patients in the
because the effect of providing home health services is not yet known and therefore cannot be used to create a profit chart, one possible proposal might be a one-month pilot program where the 60 patients (2.5%) most likely to be readmitted are given home health services (leftmost orange box).
20.4 Areas where the model struggles
It can only be concluded, therefore, that the problem lies with the data.
Internal data: In this case, it could be argued that more data on past visits should be collected for patients that are frequent visitors (per Figure 20.6).63 Are they repeat visitors for the same problem? It was also seen that predictive ability could be extracted from the very limited text in the diagnosis descriptions. Patient records contain a high volume of text that could be further mined for additional features.
External data: While it is not clear that mining public external data will be a worthwhile exercise for this project, it is likely that this project could benefit from purchased data such as grocery store purchase data, financial data, distance to hospital, and so on. It might be appropriate to argue for a budget to pilot such data.
20.5 Most predictive features
When explaining a model, directional results tend to make intuitive sense
the discussion turns to different types of features that are useful in model evaluation and presentation.
20.6 Not All Features are Created Equally
Often information that helps a model make high-quality decisions is not helpful in changing practice.
There are four kinds of features to consider before going into a management presentation:
Features that need to be changed and therefore require a re-run of the models
Features requiring further examination: Preferably examined further with the help of a subject matter expert (SME)
Immutable features: These are features that are good for modeling but are of no value to management in the event that they want to implement corrective actions.
Mutable features: These are the features that management could potentially change. Because these are the measures that may be manipulated, it is possible that by doing so, management can improve the status (health, mood, etc.) of the subjects in a given dataset.
20.7 Recommended Business Actions
Slide number nine addresses the final part of the presentation, which should 279
contain explicit recommendations for next steps. In this case, three concrete recommendations are available