Please enable JavaScript.
Coggle requires JavaScript to display documents.
Chapter 7: Decision Analytic Thinking I (Other evaluations metrics…
Chapter 7: Decision Analytic Thinking I
Problems with unbalanced classes
distribution of classes are unbalanced / Skewed
happens when looking at relatively small or rare classes in large populations
Evaluating Classifiers
Plain accuracy
Accuracy = Number of correct decisions made / Total number of decisions made
Binary Classifiers
Positive
Deserves attention
Ex. Account reacts positive for Fraudulent activity
Negative
Problems with unequal costs and benefits
no distinction between false positive errors and false negative errors
False positives and false negatives will rarely ever carry the same weight
Estimate the cost or benefit of each decision a classifier can make. There should be an expected profit (cost or benefit)
Other evaluations metrics
fundamental summaries of confusion matrix
Sensitivity = True Negative / (TN + False Positive)
Specificity = True Positive / (TP + False Negative)
Accuracy = (TP +TN) / (P + N)
Evaluations, baseline performance, and implications for investments in data
It is important to consider carefully what would be a reasonable baseline against which to compare model performance.
What is an appropriate baseline for comparison?
Key Framework: Expected Value
Decomposes Data-analytic thinking into:
The elements of the analysis that can be extracted from other sources
The elements of the analysis that need to be acquired from other sources
The structure of the problem
Becomes the weighted average of the values of the different possible outcomes, where the weight given for the outcome is the probability