Chapter 20: Communicate Model Insights (Six types information that…
Chapter 20: Communicate Model Insights
Six types information that should be communicated during a business presentation
Model quality metrics (confusion matrix)
Areas where a model struggles (potential for improvement through more data––features & cases)
Most predictive features for model building
Feature types especially interesting to management (e.g., insights into the business problem and unknowns uncovered during the modeling process)
Recommended business actions (i.e., to implement model or not, any business decisions to implement at various probability thresholds, and how will doing so change practice?)
Questions that may be asked:
What is natural language processing, really? How does a regression work?
How do you answer these questions?
It is best to answer these by saying there is not enough time to delve into fundamentals of the model
Elaborate with a simple explanation of the models: They all determine the generalizable relationship between the features and the target and place those relationships into a model that can be used to both understand those relationships and predict the outcome of cases not yet encountered.
20.1 Unlocking Holdout
Before taking a model to use for business decisions, make sure it has no mistakes
Unlock the holdout to make sure your model has no mistakes. Check the holdout and cross validation scores to make sure the holdout sample fits the model. IF the holdout sample scores are much lower than the cross validation scores, the model may have some mistakes or poor assumptions
20.2 Business Problem First
It is crucial to keep the business problem in mind throughout the data mining process.
When presenting, start with the business problem that you are solving
20.3 Pre-processing and Model Quality Metrics
Perform a quick overview explaining the process for procuring data, cleaning that data, and carefully addressing relevant issues
It may be tempting to share code or go into great details here, but it is best to restrict this data processing information to one slide without code.
To address model quality metrics, begin with the confusion matrix for the chosen model and annotate it for the audience
Explain how the model will perform on patients it has never seen before, but was trained by historical data.
Clarify the model performance may deteriorate given new data, but that this performance can be measured and the model could be retrained
Then animate Positive Predictive Value, and explain its significance
What it means is the level of precision of a model in terms of its ability to predict
20.4 Areas where the model struggles
Explain how the model has no errors, and that the data is where the model struggles.
If managers are interested in the potential insights that come from the model, then lobbying for more data should be reasonable.
20.6 Not All Features are Created Equally
Four kinds of features to consider before presenting to management
Features that need to be changed and therefore require a re-run of the models:
. As a general rule, when features need to be changed, it is often necessary to then re-run the model after making these changes. Any row with this “expired” value should be removed before generating new models.
Features requiring further examination
Preferably examined further with the help of a subject matter expert (SME). These are features about which a manager is likely to ask follow-up questions
These are features that are good for modeling but are of no value to management in the event that they want to implement corrective actions.
An immutable feature is one that a management team cannot change, such as the number of years that have passed since someone entered a given industry after receiving their bachelor’s degree
: These are the features that management could potentially change. Because these are the measures that may be manipulated, it is possible that by doing so, management can improve the status of the subjects in a given dataset.
20.5 Most predictive features
Directional results tend to make intuitive sense
Focus on the most important features to the business problem--explaining everything is unnecessary and will confuse the listener
Create a story around the most predictive feature
20.7 Recommended Business Actions
Should contain explicit recommendations for next steps, giving a number of recommendations