Please enable JavaScript.
Coggle requires JavaScript to display documents.
Evaluate Model Performance (Introduction (AutoML should know what…
Evaluate Model Performance
Introduction
AutoML should know what algorithms are doing on a technical level
acquire skill to translate
dont talk about how algorithms work but about their performance
logloss does not measure how far form the target the average is
FVE Binomial shows how much variance is equal to the R squared value
allows reorder of the leaderboard
Sample Algorithm and Model
value in understanding algorithms
Tree based algorithm example
uses python
follow steps:
find most predictive feature and put at root of tree
split the feature into groups that are homogenous
Repreat step 2 for each branch
ROC Curve
single most important screen in datarobot
ROC - receiver operating characteristics
chart has two options: validation and cross validation --> scores shouldn't be too different
green means "readmission = true"
purple means " readmission = false"
goal for algorithm is to assign high probabilities to positive cases and low probability to negative cases.
mountains overlapping means data is not the best quality
threshold line is shown to ID the best cutoff for the 2 mountains to be seperated
confusion matrix
light is green and quadrant is green = true positive (TP)
red light and red quadrant = true negative (TN)
when TN and TP are combined that provides and accurate measure aka what proportion of decisions made are correct? use TP+TN/ all cases
top right quadrant represents where algorithm incorrectly predicted a green light when it was really red --> this is a false positive
in the bottom left quadrant the algorithm predicted a red light when it was really green --> false negative
every case is evaluated and put in a quadrant
1 more item...
Using the lift chart for business decisions
lift chart sorts all validation cases by their probability of readmission
once sorted they are placed in bins
ideal to have blue and orange lines overlapping --> strong model indicator
lines will also show gaps where model needs work
export to excel for closer examination
create charts