Please enable JavaScript.

Coggle requires JavaScript to display documents.

MACHINE LEARNING ML3 (Linear Regression (t = f(x) (Vector Form ($$ t =…

- - - - $$ t = w^Tx $$
  - - - $$ L_n = \frac{1}{N} \sum_{n=1}^{N} {(t_n - w^T x_n)}^2$$
        
        $$ w $$ is a vector of the weights
        
        $$ x $$ is a vector of all the terms in the polynomial - $$ x^0, x^1, x^2 $$
        
        Identity: $$ \sum_{d=1}^{D} {a_d}^2 = a^Ta $$
        
        "The sum of the square of a vector is vector transpose times vector"
        
        $$ L = \frac{1}{N} {(t - Xw)}^T(t - Xw) $$
        
        "average" of true values minus Xw
        
        X is all the individual x vectors transposed and stacked on top of each other
        
        Since this X vector includes all the terms of the model, then this formula is the same for all models
        
        Each ROW of the vector is a different example
        
        Each COLUMN of the vector is a term in the model. The first column is usually all ones.
        
        w is the vector of weights as before
        
        $$ w = {(X^TX)}^{-1} X^Tt $$
        
        General expression to find the best weightings for a model
- - - - Based on Bayes Rule
      - Prior
        
        Used to specify e.g. if we know there are more of one class than another
        
        Specifies probabilities for each class irrespective of the data
      - Naive-Bayes
        
        Assumes the components of xnew are independent, so the likelihood can be computed as a product
        
        Easier to fit D uni-variate distributions than it is one D-dimensional one, so used for high values of D
      - Likelihood
        
        for each class: Normal distribution with mean at the centroid of the class
        
        $$ \prod_{d=1}^{D} N(\mu_{kd}, \sigma_{kd}^2 ) $$
    - - Use regular output of f(x_new; w) and use a function h() to "squash" it between 0 and 1 to be a probability
      - Sigmoid Function
        
        $$ \frac{1}{ 1 + exp(-w^Tx_{new}) } $$
      - Can do "Bayesian Logistic Regression"
        
        Choose a Gaussian prior
        
        always important from a data analysis point of view
        
        not needed "for the maths"
        
        Likelihood
        
        Assume independence between examples
        
        already defined as the "squashing" function
        
        Prior is not conjugate to likelihood
        
        Find most likely value of w - a point estimate
        
        MAP - Maximum A Posteriori
        
        g(w; X, t) is proportional to p(w | X, t) so the params that maximise both are the same
        
        We can't solve exactly, but we can do OPTIMISATION
        
        Newton-Raphson
        
        Approximate posterior with something easier
        
        Laplace approximation
        
        What defines similarity?
        
        Mode (highest point) is in the same place
        
        Similar shape?
        
        Something easy to manipulate?
        
        Find mean from MAP
        
        Use Laplace to find covariance
        
        Cannot evaluate the expected value of the results
        
        But we can sample from it, estimating the expectation with a high number of samples
        
        Sample from posterior
        
        MCMC sampling
        
        Markov Chain Monte Carlo
        
        Metropolis Hastings
        
        The algorithm that selects/creates the MCMC
        
        MH proposes a w_s given w_s-1 and then either accepts or rejects it
        
        1 more item...
        
        Treat the new proposal as a random variable conditioned on the last accepted step
        
        2 more items...
  - - - No 'training' phase
      - Binary or multi-class
      - Process
        
        Choose K
        
        Smaller K - more random ties/guessing -> more non-deterministic!
        
        Larger K - smoother class boundaries, risk of extinguishing some classes if the no. of examples in them is very small
        
        Use cross-validation, choose K with least misclassifications
        
        For a test object, find the K nearest points in the TRAINING set to it
        
        Assign the class which is most prevalent in the K selected points. Random if a Tie
    - - finds the decision boundary that maximises the margin
        
        In 2D space, it can be represented as a line
        
        $$ w^Tx + b = 0 $$
        
        we must find the optimal w and b
        
        optimize based on MARGIN
        
        Margin is the PERPENDICULAR DISTANCE to the decision boundary from a point
        
        Maximise the margin, since the further away it is from all the closest points, the better the decision boundary is, as the more centered it is
        
        $$ 2y = \frac{1}{||w||} w^T(x_1 - x_2) $$
        
        ...
        
        We want to maximise $$ y = \frac{1}{||w||} $$
        
        1 more item...
        
        Constraints
        
        if tn is 1, then decision boundary value must be >= 1
        
        If tn = -1, then decision boundary value must be <= 1
        
        Hence we use tn in {1, -1} and not {1, 0}
      - KERNELS
  - - - True Positives / (True Positives + False Negatives)
    - - True Negatives / (True Negatives + False Positives)
    - - Plot between sensitivity and 1 - specificity
      - Shows how Specificity and Sensitivity vary as the threshold value changes (i.e. the threshold where we classify as a 0 or 1)
      - The best curve is further to the top left hand corner -> perfect classification
      - Performance quantified by the Area Under Curve (AUC). The higher, the better
        
        Generally a safer measure than 0/1 loss
    - - Table with coumns the actual class, rows the predicted class, and values are TP, TN, FP, FN
      - Tells us where mistakes are made
        
        Maybe we should combine these classes?
        
        Maybe we need more data in these classes?
      - Normalizing the columns gives Se and Sp
      - Useful for multi-class classification
    - - The no of things predicted correctly / the total no. of things
- - - - Parameters
        
        Mean
        
        $$\mu $$
        
        Variance
        
        $$ \sigma^2 $$
    - - Is equivalent to sampling from N(wT x, sigma^2) since it's centered around the definite prediction
    - - this is due to sampling from a distribution based on actual data
  - - - Define probabilities for different events taking place
    - - Define a density function p(x)
        
        NOT probabilities
        
        Probability of a given range of values is the Area Under Curve of p(x)
        
        Indefinite integral i.e. total area under curve must equal 1
    - - P of X and Y happenning
      - Can think of as a 2D table
      - If X and Y are independent, then P(X, Y) = P(X) * P(Y)
      - If X and Y are dependent, then we must use CONDITIONAL PROBABILITIES
        
        P(A | B) "probability of A given B"
  - - - Therefore we want to combine them all, with a Joint Likelihood
      - Assume all are completely independent, then it's just the PRODUCT of likelihoods for all training examples
- - - - In each iteration you update the centroids/means and then re-assign points to clusters until you detect convergence
    - - Can substitute xTx terms for a kernel function
  - - - Maximize the likelihood to find the best parameters
        
        $$ p(X|\Delta) = \prod_{i=1}^N p(x_n|\Delta) $$
        
        Each cluster is independent
        
        $$ \prod_{i=1}^N \sum_{k=1}^K p(x_n | z_{nk} = 1, \Delta_k) p(z_{nk} = 1 | \Delta ) $$
        
        Un-marginalise k
        
        This is our final likelihood function
        
        Convert to log likelihood
        
        End up with Log of a sum, which is "bad"
        
        JENSEN'S INEQUALITY
        
        1 more item...
        
        Use jensen's inequality, add an arbitrary distribution over itself in front of everything, do some magic and we get a log-likelihood that works