Please enable JavaScript.

Coggle requires JavaScript to display documents.

Machine learning (Optimalization (Gradient descent (Error function = Loss…

- - - - w = w - alpha * dw(E(w)) # # #
        
        Derivation
        
        w = w - alpha Xi (Xi * w - ti)
    - - w
    - - alpha
    - - Derivation E(w)
        
        (x^T w - t) x
      - gradient w of error function
      - E(w)
        
        (x^T * w - t)
      - Examples
        
        Mean Square Error
        
        MSE
        
        Mean Squeare Error as Minimum likelihood estamation
        
        Mean Square Error as MLE
        
        MSE as MLE
        
        Root Mean Square Error
        
        RMSE
        
        root square of MSE
      - Error between predicted values and real target
    - - Too much data
      - direct optimization id not feasible
    - - (Regular) Gradient Descent
      - Online (or Stochastic) Gradient Descent
        
        SGD
      - Minibatch SGD
    - - ...
- - - - Overfitting
        
        Solving
        
        Generalizaiton
        
        L Forms
        
        L1
        
        absolute value of magnitude
        
        Lasso regression
        
        1 more item...
        
        L2
        
        squared magnitude
        
        Ridge regression
        
        Lambda
        
        Generalization gap
      - Underfitting
    - - splited to
        
        Testing set
        
        Test error
        
        Training set
        
        Train error
        
        Split algorithm
        
        Cross Validation
        
        K-Fold Cross Validation
      - Input values
        
        x1, ..., xN
        
        Features
        
        Extended input values
        
        Feature engineering
        
        Polynomial features
        
        One-Hot Features
        
        Raw observations
      - N targets
        
        t1, ..., tN
    - - Lower generalization error
      - Perform well on previously unseen data
      - Lower test error
    - - y1, ..., yN
    - - That thing that takes input and returns output
      - Need to learn
      - Capacity
    - - Parameters that are not learned by algorithm by default
    - - Bagging
  - - - Class prediction
      - Assigning one of K
    - - Models
        
        Perceptron
        
        Issues
        
        if input set not lineary separable then alg never finish
        
        cannot be easili extend t classfication with more classes than two
        
        Performs only prediction
        
        Result
        
        Class
        
        Perceptron as SGD
        
        we use loss function
        
        Result
        
        Class
        
        Logistic regression #
        
        Sigmoid function #
        
        Result
        
        Probability
        
        Multilayer Perceptron
        
        Activation function
        
        Examples
        
        ReLU
        
        None
        
        Sigmoid
        
        tanh
        
        Proofs
        
        Universal approximation theorem 89
        
        Universal Approxiamtion Theorem for ReLUs
        
        Universal Approxiamtion Theorem for Squashes
        
        Layers
        
        Hidden layers
        
        Input layer
        
        Output layer
        
        Training
        
        Can be trained using an SGD
        
        Or Mlutilayer Logistic regression #
        
        Support Vector Machines
        
        SVM
        
        Karush-Khun-Tucker Conditions
        
        KKT Conditions
        
        Variants
        
        Soft-margin SVM #
        
        Hard-Margin variant
        
        Soft-Margin SVM
        
        SVM for Non lineary separable data
        
        Hinge loss
    - - Models
        
        Multiclass logistic regression
        
        Softmax function
        
        MLR
        
        Multiclass SVM
        
        No general approach
        
        One vs Rest
        
        One vs One
        
        Naive Bayes Classifier
        
        Variants
        
        Multinomial NB
        
        Bernoulli NB
        
        Gaussian NB
        
        Mutlivariative Gaussian NB
        
        NB
  - - - Producing a number x for given input
    - - Linear regression #
        
        Weights
        
        w
        
        Bias
        
        b
        
        Sometimes is hidden as weight with value 1
        
        Types
        
        SVD
      - Kernel Linear Regression
        
        Kernel trick
        
        Decide value not by features but by examples
        
        Kernel
        
        fi
        
        Examples
        
        Gaussian
        
        RBF
        
        Polynomial
      - SVM for regression
  - - - Classification Decision Trees
        
        Criterion
        
        Gini index
        
        Entropy criterion
      - Random forest
        
        Use multiple decision trees with hiden features
      - Gradient boosted trees
      - Regression Decision Trees
    - - Maximum tree depth
      - Minimum examples to split
      - Maximum number of leaf nodes
    - - NLL
- - - - Maximum likelihood estimation
        
        Properties
        
        MLE
- - - - Iterative algorithm
    - - (Soft K Means)
- - - - Central limi theorem
      - Mutli variate Gaussian distibution