Please enable JavaScript.
Coggle requires JavaScript to display documents.
evaluate model performance (must know the algorithms at a technical level…
evaluate model performance
must know the algorithms at a technical level or know function of the algorithms conceptually
ability to understand model's performance and business context is important
if no ability to predict an outcome, measures to evaluate are fraction of variance explained and binomial
fve is how much variance in dataset can be explained or is equivalent to an r^2 value
sample algorithm and model
decision tree classifier - repeatedly find the most predictive feature at that instance and split into 2 groups that are internally homogeneous
examine models but not examine of algorithm and created them
find the most predictive feature - the one that best explains the target - and place it at the root of the trees
split the feature into 2 groups at the point of the feature where the 2 groups are as homogeneous as possible
repeat step 2 for each new branch
ROC Curve - receiver operating characterisitces
validation and cross validation scores should not differ wildly
density distribution - present each group as the same volume in case of an unbalanced target
goal in algorithms is to assign high probabilities to positive cases (readmission) and low probabilities to negative cases (non-readmission)
threshold - probability value at which datarobot changes a prediction from a negative to a positive
measure denotes model success in predicting positive values
must be static and exist at a single threashold
create a confusion matrix - 2 by 2 - what it should have done vs what it actually did and the detection of such
false positive - saw red but algorithm guessed green
false negative - light was green but should have been red
positive predictive value - derived from 2 rightmost quadrants - how often model is correct
true positive rate - sensitivity - measures proportion of positive cases correctly identified
matthews correlation coefficient - correlation measure that is a strong indicator even when target is unbalanced
measured as area under the curve value
using the lift chart for business decisions
constructed by sorting all validation cases by probability of readmission then split into bins
enable drill down - to download actual predictions
remove holdout samples and sort
create a profit chart that shows which percent location of dataset is optimal project investment
actual vs prediction shows how off (over or under predicted) the data