Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapters 17 and 18 (Evaluate Model Performance (confusion matrix (traffic…
Chapters 17 and 18
Evaluate Model Performance
LogLoss
doesn't constitute a reasonable measure of how far from target predictions are at the level of average case
Fraction of Variance Explaied (FVE) Binomial
sense of how much of the variance in the data set has been explained and is equivalent to an r^2 value
metric states how far off, % wise the model is
decision tree classifier
tree-based algorithms
repeatedly finds the most predictive feature at an instance and split it into 2 groups that are as homogenous as possible.
ROC Curve
change does not have a large impact = good thing
confusion matrix (traffic lights)
generally set up for binary problems
these 2 quadrants constitute a correct classification and measure: accuracy
what proportion of the model is correct?
on the left, what the car should've seen
the top shows the actual detection made
bottom right: true positive
the light green, algorithm determined it to be green
top left: true negative
the light red, algorithm determined it is red
top right: false positive
light was red, algorithm thought it was gree
major failure
bottom left: false negative
light is green, algorithm thinks its red
every case is evaluated and placed in one of the quadrants
the distinction between TP and TN is what the system constituted as a positive
DataRobot treats the lowest value as the negative and highest as positive
true, yes and 1 = positive
false, no and 0 = negative
recognize the case-by-case importance of the kind of mistake the model makes
Positive Predictive Value (PPV)
#
derived from the 2 rightmost quadrants
Precision
how often the model is correct when it indicates that something is positive
Measures
True Positive Rate TPR
sensitivity
number of cases in bottom right/ number of cases in the 2 bottom quadrants
Negative Predictive Rate NPV
mirror image of PPV
True Negative Rate
specificity
False positive Rate
fall out
suggests that a % of negatives cases are reported as positives
leads to high costs
Comparing Model Pairs
model comparison
allows the selection of 2 models from the leaderboard
prioritizing criteria and selecting a model
predictive accuracy
prediction speed
speed to build model
familiarity with model
insights