Please enable JavaScript.

Coggle requires JavaScript to display documents.

Lecture 3 Classifiers (Classifier algorithms (Techniques) (k Nearest…

- - - - Building a classifier
        
        New Data
- - - - If previously unseen example is above the line
        
        Then class is red
        
        Else class is blue
      - Training (Intuition)
    - - Perfect
      - Useless
        
        There is no way to draw a straight line to separate blue from red
      - Reasonable
    - - Fast to train
      - Fast to predict
      - Low risk of over-fitting
    - - Primarily suited for numerical data
        
        No easy way to handle categorical attributes
      - Challenges with high dimensional data
    - - There is no simple linear classifier implementation in WEKA
      - A close alternative may be used
        
        Logistic regression
        
        WEKA -> functions -> SimpleLogistic
  - - - If it walks like a duck, quacks like a duck, then it's probably a duck
    - - If k=3 (neighbours to look at)
        2 blue neighbours, 1 red
        Prediction: (?) is blue
    - - New example arrives
      - Compute the distance from the new example to each training example
      - Identify k nearest neighbours
      - Use class labels of the nearest neighbours to determine the class label of the new example
    - - Store training data
      - Chose a distance measure
    - - For numeric attributes
        
        Manhattan distance (Taxicab/cityblock)
        
        Cumulative distance between each attribute
        
        Example
        
        Point 1: (1, 2, -1)
        
        Point 2: (0, 1, 0)
        
        Manhattan distance between P1 and P2 is 3
        
        Euclidean distance
        
        Straight line distance between two points
      - For categorical distance
        
        Hamming distance
        
        Number of positions where the symbols are different
        
        Example
        
        Point 1: (sun, holiday, blue)
        
        Point 2: (rain, holiday, red)
        
        Hamming distance between P1 and P2 is 2
        
        2 attributes are different in example
    - - For a data-set of N examples we can build a square NxN symmetric matrix of pairwise distances
      - \[ \text{Euclidean:} \\ d(x, y) = \sqrt{\sum^{m}_{i=1}(x_{i} - y_{i})^2} \]
      - \[ \text{Manhattan / city-block:} \\ d(x, y) = \sum^{m}_{i=1}|x_{i} - y_{i}| \]
      - \[ x = [x_{1}, x_{2} \cdots , x_{m}] \\ y = [y_{1}, y_{2}, \cdots , y_{m} \]
    - - Depending on units of measurement, we may get different nearest neighbours
        
        One solution is to normalise the data to pure numbers
        
        E.g. standardisation: x = (x - mean(x) / std(x)
        
        In WEKA: (Pre-process) filters -> unsupervised -> attributes -> standardise
    - - Simple to implement
      - Interpretable (white box), prediction mechanism is easy to understand (depending on dimensionality of data)
      - Can model complex decision boundaries (non-linear)
    - - Poor performance on high dimensional data (many attributes)
      - Very sensitive to irrelevant features
      - Slow classification time for large data-sets
        
        Needs to compute distance to every training example
    - - Lazy -> iBk
      - Main parameters
        
        KNN
        
        How many neighbours to look at
        
        If you are working with large data-sets you can speed up the KNN classifier by "throwing away" some data
        
        E.g. delete all examples that are surrounded by members of their own class
        
        Data editing
        
        E.g. via clustering
        
        CrossValidate
        
        Selects good k automatically
        
        nearestNeighbourSearchAlgorithm
        
        You can change distance functions
  - - - Predicting
        
        Go down the decision tree from the top to bottom until you reach the label
      - Training
        
        A tree is constructed in divide and conquer manner
        
        Calculate how much information about the label is provided by each attribute
        
        Split at the most informative attribute
        
        Continue until all nodes are pure (all labels are the same in a node), or pre-defined depth is reached
        
        If the resulting tree is very big, pruning is used, i.e. some little informative nodes are cut
      - Example
      - What can be solved by decision tree classifiers?
        
        Very difficult
        
        Easy
        
        Very easy
      - Advantages
        
        Interpretable decision (domain experts love them)
        
        Easy to generate rules for decision making
        
        Relatively fast training
        
        Fast decision making
      - Limitations
        
        Prone to over-fitting
        
        May be hard to control
        
        The model grows with number of training examples
        
        Pruning is necessary
      - WEKA
        
        Classifiers -> trees ->
        
        J48
        
        Most popular type of tree
        
        Special splitting criteria
        
        Main parameters
        
        Can switch on-off pruning
        
        Allow two or more nodes per split
        
        Decision stump
        
        A simple tree algorithm with only one split
        
        Random tree
        
        Uses less attributes
        
        Based on J48
        
        Random forest
        
        Combines many trees
        
        You can choose numTrees
        
        Based on J48
  - - - Interpretable
        
        Easy to explain how the prediction was made
    - - If data-set is large, a lot of rules may be needed
      - May be prone to over-fitting
    - - In WEKA, a number of rule based classifiers are available under:
        
        Classifiers -> rules
    - - If outlook = sunny and humidity = high then play = no
        If outlook = rainy and windy = true then play = no
        If outlook = overcast then play = yes
        If humidity = normal then play = yes
        If none of the above then play = yes
  - - - Mathematical function
    - - Data is transformed to a new representation where separation (classification) becomes easier
    - - Support vector machines are useful when it is important to separate well examples that are similar to each other but belong to different classes
      - Often used for text classification
      - SVM may be linear as well as non linear
    - - Maximum distance to the classifier for the closet examples
    - - Very powerful
        
        Can model complex decision boundaries
    - - Non interpretable (black box)
      - Very slow to train if training set is large (many examples)
      - Suited for numerical attributes
        
        No easy way to work out categorical
    - - There is no direct implementation in WEKA (would be very slow)
      - There is a good library LIBSVM (in C and java)
        
        Plug-ins for major data mining software platforms
  - - - Powerful (complex decision boundaries
      - State of the art in many domains
      - Deep learning
    - - Non intuitive decision making (black box)
      - High risk of over-fitting
      - Many parameters to tune
      - Typically for numerical input attributes
      - May be slow to train
    - - Classifiers -> functions -> MultilayerPerceptron
  - - - Probabilistic classifier
      - Mathematical function to compute the probability that a new example belongs to a certain class
      - Assign the class label that had the highest probability
    - - Compute probability functions
    - - It assumes that input attributes are independent from each other, that is often not realistic
    - - Can work with little training data
      - Primarily for categorical attributes, but easily handles numerical as well
    - - Assumes that input attributes are independent
    - - Classifiers -> bayes -> NaiveBayes
      - Bayes Network
        
        Classifiers -> bayes -> BayesNet
        
        A related more powerful probabilistic classifier