Please enable JavaScript.

Coggle requires JavaScript to display documents.

Tree-based Learning, Weakest Link Pruning, Boosting pt.II, Bootstrap…

- - - - 2) Calculate Weighted average of leaf node impurities of the split
      - 1) Calculate Gini impurity for each variable; 1 variable=1 split
    - - 1) Sort the values in the variable in ascending order
      - 2) Calculate average value for all adjacent values
      - 3) Calculate the Gini impurity for each average value; <avg value= 1 split
      - 5) Take the 'average' with minimum Gini impurity and split the data on it
      - 4) Calculate Weighted average of leaf node impurities of the split
    - - 1) Calculate Gini impurity for all the discrete values except the largest value in the variable; <d1 = 1 split
      - 2) Calculate Weighted average of leaf node impurities of the split
    - - 1) Calculate Gini impurity for all values and all possible combinations except (V1 or V2 … or Vn) where n=number of values in the variable; V1 (or V2..) = 1 split
      - 2) Calculate Weighted average of leaf node impurities of the split
- - - - H(E)=−∑c(j=1) (pj logpj)
      - Less Entropy = More Homogeneity
      - Information gain = 1-Entropy ?
      - Information gain of decision split
        
        e.g.
        information gain = 0.97095 - 12/20 0.65002 - 8/20 1
    - - probability that a randomly chosen sample in a node would be incorrectly labeled if it was labeled by : the distribution of samples in the node
      - Less Impurity = More Homogeneity
      - G = ∑C(i=1) (pi (1-pi)) where C=number of classes
        (p^2+q^2 or 1-p^2-q^2)
      - Gini impurity = 1 - Gini
        n = number of classes
        
        Cos Gini Impurity of Split on Class < Split on Performance => Split on Class :check:
  - - - ∑((X-X¯)^2)/n where X¯=mean, n=number of values
    - - RSS = ∑(j=1->J)∑(i(-Rj) (yi-yRj^)^2
- - - - Take (nth,n+1th) smallest observations of the predictor variable (n=1 initially)
      - Take average of them & use that "average" as node to split for the tree with average value of observations on the left (observations < "average") as left child and average value of the observations on the right (observations < "average") as right child
      - Calculate the residual sum of squares RSS=(yi-y^i)^2 for all observations
      - Plot RSS against the threshold values of the predictor variable ("average")
      - Set n=n+1
      - Repeat Steps 1-5 until n==N (N=number of observations)
      - Take the threshold with minimum RSS and split the tree.
      - Repeat from 1-7 until num of observations in each leaf < min number of observations required to split the node.
    - - For each variable, do the steps of 1-7 of single variable algorithm and take the node with min RSS as candidate for node(i)
      - Compare RSS for each of the candidate and take the node with lowest RSS as the node(i) of tree.
      - Grow the tree doing steps 1-2 until num of observations in each node(i)/(leaf) <= min num of observations required to split the node
- - - - n_estimators
      - criterion{“gini”, “entropy”}
      - max_depth=None
      - min_samples_split=2
      - min_samples_leaf=1
      - max_features='auto'
      - max_leaf_nodes=None
      - bootstrap=True
      - oob_score=False
  - - - Class of most votes of all the trees
- - - - Initial prediction = 0.5. Calculate Residual= Actual-Prediction for each observation.
      - XGBoost Tree for Regression
        
        1) Each tree starts out as single leaf w. all the residuals
        
        2) Calculate Quality/Similarity Score for parent root node;
        
        Similarity score=Squared Sum of all residuals in the leaf / (Num of residuals + λ)
        Where λ=regularization parameter, default=0;
        
        3) Take (nth,n+1th) smallest observations of the predictor variable (n=1 initially) of the parent node to be split
        
        4) Take average & use that "average" as a condition (<"average") for the node to split with residuals on the left (observations < "average") as left child and residuals on the right (observations < "average") as right child
        
        5) Calculate SS for each leaf
        
        6) Calculate Gain of tree.(w. split <"average")
        
        Gain=SS left child + SS right child – SS root
        
        7) n=n+1
        
        8) Repeat 3)-7) until n==N where N=total number of observations in the parent node to be split
        
        9) Use threshold (<"average") that gives the largest GAIN as the branch in the tree
        
        10) Repeat 3)-9) until it reaches max depth of tree (default=6 levels) OR min num of samples to be split [Cover]
      - Calculate prediction for each sample.
        
        Output/prediction of each leaf = Sum of residuals of all samples in leaf/(num of residuals+λ)
        
        Prediction for each sample= 1st tree prediction + Σj=2->(i-1) (lr x jth tree prediction) + (lr x current tree prediction)
        where ith tree=current tree; lr = learning rate (default=0.3)
      - Repeat #1-#3 until max num of trees OR residuals are super small.
    - - Initial probability = 0.5; Initial prediction= ln(p/1-p) = 0;
        Calculate Residual = Actual - Probability; Event happening = Actual = 1;
      - XGBoost Tree for Classification
        
        1) Each tree starts out as single leaf w. all the residuals
        
        2) Calculate Quality/Similarity Score for root node
        
        Where λ=regularization parameter, default=0;
        
        3) Take (nth,n+1th) largest observations of the predictor variable (n=1 initially) of the parent node to be split
        
        4) Take average of them & use that "average" as a condition (<"average") for the node to split with residuals on left (observations < "average") as left child and residuals on right (observations < "average") as right child
        
        5) Calculate SS for each leaf
        
        6) Calculate Gain of tree (w. split "<average")
        
        Gain=SS left child + SS right child – SS root
        
        7) n=n+1
        
        8) Repeat 3)-7) until n==N where N=total number of observations in the parent node to be split
        
        9) Use the threshold (<"average") that gives the largest GAIN as the branch in the tree
        
        10) Repeat 3)-9) until it reaches max depth of tree (default=6 levels) OR min num of samples to be split [Cover]
      - Calculate prediction for each sample
        
        1) Calculate output / prediction of each leaf in tree
        
        e.g. for leftmost leaf, output = -0.5 / (0.5 * 0.5) = -2
        
        2)Prediction for each sample= 1st tree prediction + Σj=2->(i-1) (lr x jth tree prediction) + (lr x current tree prediction);
        where ith tree=current tree; lr = learning rate (default=0.3);
        
        3) Probability = e^(prediction)/(1+e^(prediction))
      - Repeat #1-#3 until max num of trees OR residuals are super small.
    - - Tree Complexity
        
        Pick a number for γ (Tree complexity parameter)
        
        :arrow_double_up: λ=> :arrow_double_down: SS => :arrow_double_down: Gain => :arrow_double_down: Pruning Threshold
        
        If ancestor parent node<γ but descendent parent node>γ, keep the ancestor parent node
        
        If gain of the parent node > γ, keep that parent node. If not, remove/ prune that parent node/ that split
      - Cover
        
        Set Cover=0 (Classification)
        
        What?
        
        denominator of SS minus λ
        
        2nd derivative of loss function
        
        min_child_weight
        
        Min num of Residuals in each leaf to split
        
        default=1(:check: Regression)
  - - - Prediction for each sample = 1st tree prediction + Σj=2->(i-1) (lr x jth tree prediction) + (lr x current tree prediction) ;
        where ith tree=current tree; lr = learning rate;
      - Prediction of each leaf
      - Probability = e^(pred)/(1+e^(pred))