Please enable JavaScript.

Coggle requires JavaScript to display documents.

Lecture 5 Regression, Ensembles (Evaluation of Regression Models…

- - - - A predictive model for regression task
      - Algorithm that implements regression
      - Maps input data to a target
  - - - Predicting
        
        Mathematical function
        
        e.g. selling price = 10 * house size + 51
        
        10 and 51 are results of training
      - Training (intuition)
        
        Minimise the sum of squared distances from the examples to the regression line
      - What can be solved by linear regression models?
        
        Perfect
        
        Not good
      - Advantages
        
        Fast to train, fast to predict
        
        Low risk of over-fitting
      - Limitations
        
        Can model only simple relations
        
        Poor performance if input attributes are correlated with each other
      - WEKA
        
        Classifiers -> functions -> SimpleLinearRegression
        
        Missing values are not allowed; can only deal with numeric attributes
        
        Keep in mind this is not a classifier
        
        Classifiers -> functions -> LinearRegression
        
        Can eliminate influence correlated attributes (ridge parameter)
    - - k nearest neighbour for regression
        
        Prediction
        
        Finds k nearest neighbour, takes an average of their targets as a prediction
        
        In WEKA
        
        Classifiers -> lazy -> iBk
      - Regression trees and Rule based regression models
        (operation is similar to trees and rules for classification)
        
        In WEKA:
        
        Classifier -> trees -> M5P
        
        Set the option buildRegressionTree = true
        
        Classifiers -> trees -> DecisionStump
        
        Simple tree, one level
        
        Classifiers -> rules -> DecisionTable
      - Neural networks
        
        Operation is similar to NN for classification
        
        In WEKA:
        
        Classifiers -> functions -> MultilayerPerceptron
      - Quadratic and higher order regression models
      - SVM Regression
        
        Operation is similar to SVM for classification
        
        In WEKA:
        
        Classifier -> functions -> SMOreg
      - Quadratic (higher order) regression models
        
        Similar to a linear regression, but more complex model form is assumed
        
        Prediction: a mathematical function, coefficients are learned during training
        
        WEKA: no direct implementation, but it is possible to build indirectly by input attribute transformation
- - - - \[ \text{MAE} = (|-2| + 2 + |-2| + |-2|) / 4 = 2 \]
      - \[ \text{RSME} = \sqrt{((-2)^2 +2^2 + (-2)^2 + (-2)^2) / 4} = 2 \]
    - - \[ \text{MAE} = (|-1| + 1 + |-1| + |-5|) / 4 = 2 \]
      - \[ \text{RSME} = \sqrt{((-1)^2 + 1^2 + (-1)^2 + (-5)^2) / 4} = 2.6 \]
      - RSME punishes large errors
  - - - \[ \text{RAE} = (|-9| + 6 + 1 + |-2| + |-7|) / (53.6 + |-53.6| + |51.3| + |-6.4| + 14.6 + |-9.6|) = 0.18 \]
- - - - Specialisation
      - Increasing model stability, preventing over-fitting
  - - - How its works
        
        Train: multiple (k) models on different samples (data splits) and average their predictions
        
        Predict: by combining the results of k models
        
        Specialisation: each expert solves the same task, but has a different experience due to different training data examples
      - How to chose training data for each expert?
        
        Randomly select examples from all the training data
      - In WEKA:
        
        Select the inner model (e.g. decision tree) as a parameter
        
        Applicable for classification or regression tasks
    - - How it works
        
        Train: multiple (k) models on different attribute subsets and average their predictions
        
        Predict: by combining the results of k models
        
        Specialisation: each expert solves the same task, but has a different experience due to different training attributes
      - How to chose training data for each expert
        
        Randomly select attributes from all the training data
      - In WEKA:
        
        Classifiers -> meta -> RandomSubSpace
        
        Chose the inner model as a parameter
        
        Typically used for decision or regression trees
    - - How it works
        
        Train: multiple (k) models using different algorithms (e.g. Decision tree, neural network, Naïve Bayes, ...)
        
        Predict: by combining the results of k models
        
        Specialisation: each expert solves the same task, but has a different experience due to different model behind
      - How to chose training data for each expert?
        
        Each expert uses all training data
      - In WEKA:
        
        Classifiers -> meta -> Vote
    - - How it works
        
        Train: iteratively train multiple models on different samples (data splits) uses weighted voting for the final decision
        
        Predict: weighted voting by multiple models
        
        Specialisation: in boosting each new model is influenced by the performance of those built previously
        
        Boosting encourages new models to become experts for examples handled incorrectly by earlier ones
      - In WEKA:
        
        Classifier -> meta -> AdaBoostM1
        
        Only for classification
    - - Simple voting
        
        Classification - > majority vote
        
        Regression -> average
      - Weighted voting
        
        Assign higher or lower importance to each expert
        
        e.g. a senior judge can have two votes