Please enable JavaScript.
Coggle requires JavaScript to display documents.
Error Analysis - Coggle Diagram
Error Analysis
Carring out error analysis (cat classifier example)
Get ~100 mislabeled dev set examples
Count up how many are dogs
If 5 if these 100 examples are dogs the error might go down form 10 to 9.5%
if it is 50% the error could reduce to 5%
Evaluate multiple ideas in parallel (table)
dogs/cat
Great cats (lions/panthers...)
Improve performance on blurry images
For this case, I should count the cases of images selected to manually inspect, for each one of the 3 cases. This % gives a sense of what is the best path to improve the model.
build your first system quickly and then iterate
Set up a dev/test and metric
Build the initial system quickly
Use Bias/variance and Error analysis to prioritize next steps
Doesn't apply if
Have experience on problem
If exists documentation and research about the same problem
Cleaning up incorrectly labeled data
Incorrectly label is data that was labeled by human incorrectly
Training set
DL algorithms are quite robust to random error in the training set. If the error is reasonable random it ok leave that way. If the % of error is not too high.
DL are least robust to systematic errors
If the incorrectly labeled examples are a problem on my dev/test set include the column Error Analysis to count the number examples are incorrect
Only should fix the incorrect label example if it makes a significant difference on ability to evaluate algorithms on dev/test set.
Dev/test set
Apply the same process in dev and test to guarantee that they continue to came from the same distribution.
Consider examine examples your algorithm got right as well the one is get wrong.
Train and dev/test data may came from sightly different distributions.