Please enable JavaScript.

Coggle requires JavaScript to display documents.

Neural Network - Coggle Diagram

- - - - PRS-PRMLS-Day1b-Slide44
    - - PRS-PRMLS-Day1b-Slide45
  - - - PRS-PRMLS-Day1b-Slide49
  - - - Weight initialisation to small random values say between -1 to +1
    - - Vary from big to small as the model iterates
    - - So the model will try to reduce the weight
  - - - Normalisation is good for NN because it may learn faster for some inputs and not for others if the scale of the inputs are different
- - - - Output size PRS-PRMLS-Day2PM-Slide29
    - - Output size (PRS-PRMLS-Day2PM-Slide35)
    - - Max pooling's number of parameters is 0
  - - - With RNN as we perform more recurrent operation, it is equivalent to adding more layers, so there's vanishing gradient
        
        RNN cannot remember so much text!
      - number of parameters PRS-PRMLS-Day3_2-Slide21
    - - number of parameters PRS-PRMLS-Day3_2-Slide26
      - Can do feature extraction with convolution and pooling first PRS-PRMLS-Day3_2-Slide49
        
        Conv1D
        
        Output size PRS-PRMLS-Day3_2-Slide50
        
        Number of parameters PRS-PRMLS-Day3_2-Slide51
        
        MaxPooling1D
  - - - Simple, for single input/output
        
        PRS-PRMLS-Day3_1-Slide31
        
        no branching or merging, no reuse layer, no shared layer
    - - Multi input/output
      - Multi input for say a video and a question inputs
      - Multi output for instance a segmented image and an adjusted image output
      - Shared layer for example when checking a person's image against image in db
      - PRMLS-Day3_1-Slide40
    - - Max flexibility
  - - - How to train the earlier layers well
        
        Pick the right activation function
        
        Sigmoid functions can lead to vanishing gradient because the derivative is max 0.25
        
        ReLU is good for update and prevent vanishing gradient
        
        Sigmoid typically used in later few layers
        
        Init weights well
        
        Can't be too small as signal shrink and activation function leads to too small output across the network. As the output reaches later stages, the activation output will be almost zero
        
        Can't be too large if not the signal will saturate the neurons and network will become unstable, e.g., swinging to 1 very early but in later iteration swing to zero just with a little change in input
        
        Xavier Initialisation
        
        PRS-PRMLS-Day4-Slide13
        
        Was designed for sigmoid and tanh
        
        He Initialisation
        
        Designed for ReLu
        
        PRS-PRMLS-Day4-Slide17
        
        Truncate the very large and small weights
        
        Restrict the weights
        
        Large weights make the output swing with minor changes in values
        
        L2 regularisation
        
        Make the cost function take into account weight, so that the model will try to make weight small
        
        L1 regularisation will make the model make weight 0 if possible
        
        Leads to sparse weights
        
        PRS-PRMLS-Day4-Slide24
    - - So that it will not be accomstant to a unique distribution
      - PRS-PRMLS-Day4-Slide39
      - Typically batch normalisation before activation
    - - In very deep network, degradation happens
        
        So we need the net to at least perform identity mapping so it is not worse off than shallow net
        
        PRS-PRMLS-Day4-Slide50
    - - PRS-PRMLS-Day4-Slide81
    - - E.g., translation, zoom, flip, rotate
      - PRS-PRMLS-Day4-Slide85
    - - PRS-PRMLS-Day4-87
- - - - Unsupervised portion is the finding of centres of cluster e.g., using Hard C-means clustering or K-means
    - - In MLP, neurons try to capture specific features. In RBF, the neurons try to capture the similarity of training example with its centre
  - - - Pattern layer is similar to hidden node in RBF, computes distance between weight vector and input vector
      - Summation layer just add all the weighted inputs from patter layer (but in different ways)
        
        PRS-PRMLS-Day2AM-Slide17
      - Output just divide output from type A summation unit to type B summation unit
        
        Smoothening constant (PRS-PRMLS-Day2AM-Slide18)
    - - Memory-based, one pass
      - Self-growing, will grow by one pattern unit for each new training patter
    - - Converge to true regressing surface with increasing samples
      - Work in real-time and on sparse data
      - Fast
    - - Computationally heavy
  - - - In between the output neurons, there's some kind of recurrent feedback
      - m output units for each class
    - - PRS-PRMLS-Day2-Slide30
  - - - Two stage process - propagate signals forward and errors backward
- - - - PRS-PRMLS-Day1-Slide35