Please enable JavaScript.
Coggle requires JavaScript to display documents.
Evaluate Model Performance (Log loss (Great job optimizing overall success…
Evaluate Model Performance
One of Auto ML criteria
Understanding and learning
Should improve users understanding of the problem by providing visualizations of interactions between target and features
Without understanding
Auto ML not in place to predict results of analysis
In Data robot no way to know prior which algorithm will be elevated to the top of the leaderboard
So either need to
Know all algorithms at technical level
Know functions of algorithms conceptually
Business leaders: good to have general understanding of how algorithms work
In order to be more effective in model creation and validation
Log loss
Great job optimizing overall success
Does not constitute a reasonable measure
FVE Binomial
How much variance in the dataset is explained
How far off percent wise the model is from fully explaining
Decision tree
Model's blueprint
scikit-learn decision tree algorithm
Split the feature into two groups at the point of the feature where
the two groups are as homogenous as possible.
Repeat step 2 for each new branch (box)
Find most predictive feature and place at root of tree
ROC Curve
Receiver Operating Characteristics
Orange oval: cross-validation score and pull down to validation score
Predictive distribution
Density distribution (frequency)
Confusion matrix
Every case validated and evaluation and placed in quadrant
DataRobot sorts target features alphanumerically, and
treats the lowest value as the negative and the highest value as positive
Lift chart
Constructed by sorting all validation cases by their probability