Please enable JavaScript.

Coggle requires JavaScript to display documents.

Computer Vision (Conv Architectures (AlexNet (components (dropout (another…

- - - - Feature Sharing
        
        Features are same. Given enough data, the first Conv. layer of any model learns the same stuff.
      - Transfer learning
        
        You can train on many datasets for the same problem
        
        Can take pretrained individual layers from one network with different structure and simply put them into the new network, and continue the network = fine-tuning. And it works extremely well
        
        So, I can take all the knowledge that I extracted from one model and push it to another model
        
        It creates a paradigm where learning from few labeled examples is possible
      - Regularization
        
        Assuming one of the tasks is to learn normals in the image, and another is to learn the depth, and the third is to learn semantic labels per pixel (wall,floor,door,bed). The fact that all the norms point in the same direction, is a huge boos in telling you wether it's a floor or not
      - Generalization
        #
        
        Bacause of that, you get these generalizations, that allow you to create a manifold where as you move and thange the parameters of the network, the minima doesn't change, you're still in the global minima #
        
        If you do it without this generalization of solving all at once, you get this very peacky manifold, with small variations across X or Y or millions of other dimensions, which quickly push you out of the local minima and lead to a degenerate solution
      - Reused Compute
        
        4 types of computations required in DL
        
        # of output values that your network produces
        
        # of values that you read from memory
        
        # of times you write to memory
        
        # of times you Compare, so that you can actually update your gradients
        
        As you go deeper into the network, the amount of computation you have to do reduces exponentially
        
        It means that you you have a Trunk of shared convolutions, if you share those up to layer 3-4, than at the end you can add as many individual pathways as you want (10...100), and the computation of the network will never increase beyond the log scale
        
        Efficient Scaling
        By the time you get to the 2-3 Conv layer, you've already done 75% computations you need in the network
      - Efficient to scale to other tasks
      - Manageable training
    - - Pre-training. Trunk
        
        Each individual model with the given data (all individual models have the same "trunk")
        
        Trunk has to be trained with the biggest dataset that you can find: ImageNet
        
        Shared Trunk - a set of Conv layers that are shared across all tasks
        
        Each individual model is trained with the given data
      - MultiTask Training
        
        Then, much much smaller pathways T1,...Tn, that correspond to specific tasks
        
        All have different labels and different properties that they intend to describe
        
        The Trunk is fine-tuned on all datasets
      - Normalization
        
        Employ Rolling Batch - keep track of the number of exampls given to a given task. Otherwise the network will overfit to the specific task.
        
        Domain specific Batch Normalization (scaling (s,b) and moments (mean and variance)) are a MUST
        
        Or domain agnostic instance initialization
      - Loss
        
        Finally, all these pathways merge into a single loss - a weighted summation across all the losses, plus a regularization term of this Trunk
  - - - Increasing compute - training on multiple machines:
        GPU clusters / inference on embedded platforms
      - Use synthetic data for training to avoid unsupervised learning
      - MultiTask Learning for regularization - the only paradigm known today that allows you to solve multiple tasks at once
- - - - AlexNet, 8 layers, 16.4%, '12
      - VGG, GoogleNet, 19-22 layers, 6-7%, '14
      - ResNet, 100+ layers, 3.57%, '15
      - Shallow, 25-28% error, '10-'11
    - - Shallow (HOG, DPM), 34% accuracy
      - AlexNet (RCNN), 8 layers, 58%
      - VGG (RCNN), 16 layers, 66%
      - ResNet (Faster RCNN), 101 layers, 86%
  - - - Raw pixels => classifier
        
        Unlikely to work
      - + extract edges
        
        Invariant to some factors: colors, illumination, some noise
      - edges+histogram (SIFT / HOG)
        
        local histograms of orientation of edges
        
        this type of histogram can be thought of as a kind of normalization and pooling operations, locally operated on edges
      - + some bag-of-word model
        
        using k-means, or sparce coding, or FV/VLAD, => classifier
        
        state-of-the-art models before the prevalance of DL
    - - Conv, Normalization, Pooling
      - Repeat elementary layers, going deeper
      - Richer solution space
      - End-to-end learning by BackProp
- - - - LeNet-style backbone, plus:
      - ReLU
        
        reason for recent revolution in DL
        
        it can accelerate training because of better backprop, versus typical activations (tanh, sigmoid)
      - dropout
        
        another key component
        
        in-network ensembling, essentially
        
        reduce overfitting
        
        can be replaced by BatchNorm in some later network
      - data augmentation
        
        another key component
        
        label-preserving transformation, so you can do some rendom cropping/scaling/flipping, which can create more data
        
        also helps ot reduce overfitting
  - - - Conv
        
        Locally connected
        
        Spatially shared weights - a key in DL
      - Subsampling
        
        Nowdays are doing with Pooling, or just some straight Convs
      - Fully-connected layers - output
        
        We can think about the last layer as a linear classifier
      - Backprop
        
        end-to-end training by BackProp
  - - - Modularized design
        
        key component
        
        just stack a lot of 3x3 Convs as the modules following some simple rules
        
        In the same stage of NN all the layers have the same shape
        
        When the spatial size is reduces, they just increase the number of filters, to roughly keep the computation for each module (1/2 spatial size => 2x filters)
        
        So when we go deeper and deeper we don't need to design a new layers, we can just use the same templates
      - Stage-wise training
        
        When was just published
        
        VGG-11 => VGG-13 => VGG-16
        
        Nee a better initialization - that can train from scratch
        
        Not practical, because time consuming, and no end-to end
  - - - Multiple branches
        
        e.g. 1x2, 3x3, 5x5, pooling
      - Shortcuts
        
        standalone 1x1 conv in the original inception block, merged into the other branches by concatenation in the original inception
        
        have been used in all the following generations of Inception
        
        these shortcuts somehow help with optimization of this very complicated designs of GoogleNets
      - Bottlneck
        
        For each inception block there is some 1x1 Convs, to reduce the number of channels before doing expensive 3x3 and 5x5 Convs
        
        This is a kind of Bottlneck representation in terms of the number of channels
        
        In the original GoogleNet, because after 3x3 or 5x5 Convs the channels are concatenated, so usually they don't need to do some dimension increase in that case
    - - More templates, but the same 3 main properties are kept #
      - They are keep components in the success of the design of GoogleNets
- - - - No, any a.e. differentiable transformation works
    - - About twice FPROP (need to compute gradients w.r.t. input and parameters at every layer)
    - - Copy <=> Sum
- - - - Putting nearby signals together forming more complex structures and able to learn them
  - - - BatchNorm / ReLU
        
        Normalization Layer: Batch Normalization
        
        Once you get an output from Conv, you want to make sure that your features are normalized to 0 mean and unit variance
        
        Normalize distribution of each input feature in each layer across each minibatch to N(0,1)
        
        This is very simple and just allows you to learn faster. Because if distributions are of very high difference in scale, than it'll take forever to learn from examples that are high magnitude versus the low onces.
        
        Learn the scale and shift
        
        mini-batch mean
        
        mini-batch variance
        
        normalize
        
        scale and shift
        
        Again, every operation in the deep network needs to be defferentiable. BatchNorm is also different. through via chain rule
        
        Nonlinearity - ReLU #
      - Pooling
        
        Allows you to retain translation invariance. i.e. robustness to the exact location of the features
        
        Backprop through a pooling layer
        
        Most efficient approach: take any one arbitrary. Surprisingly it kinda works, if you give it enough examples, it learns to statistically find a way to find the right activation
        
        Next best approach: .....watch video again
      - Conv
        
        Share the same parameters across different locations (assuming input is stationary). If a pattern repeats itself, then you'd use the same activation (or the same filter). But if id doesn't, you'd learn a new filter
        
        That way the dimensionality of your learning problem will only increase with the complexity of the input rather than with definition of your model itself
  - - - It's proportional to the number of filters and depends on the stride. It's deterministic, and is easily computed.
    - - Also empirically. In general is proportional to the dimentionality of the problem - simple classification or a lot of classes.
      - Usually, there are more output feature maps than input feature maps. Conv layers can increase the number of hidden units by big factors (and are expensive to compute)
    - - Depends on the stride between the pools
      - Computational cost is proportional to the size of the input (negligible compared to a conv. layer???)
    - - Answer again comes with practice. In general you don't want to pool too much, because you average too much.
      - It depends on how much "invariant" or robust to distortions we want the representations to be
      - It's best to pool slowly (via a few stacks of conv-pooling layers). The smallest amount of pooling is 2x2.
  - - - Given a small mini-batch: F-PROP => B-PROP => Parameter update through GD
- - - - Done using notion of "deep metric learning"
    - - Mask R-CNN network - very high accuracy
    - - By extrapolating this problem to other problems, you can see how by solving this you can solve many many articulated problems in CV
  - - - Something that allows you to not overfit
    - - When you learn on one task, but then you perform your inference on another task
    - - same thing
    - - There is a new class of generative models that is an example of unsupervided learning
    - - Andrew's work. #
      - Ex: Perception tasks that are solve in isolation
    - - a number of optimization techniques
    - - RNN
    - - Currently we move away from engineering features to learning them. But now we are engineering networks, which is not the best solution
      - New works: people write algorithms that actually generate NN architectures from data
- - - - Is it appopriate for the task?
      - Does it have degenerate solutions? Check "pull-up" term
- - - - a deeper model has a reacher solution space, and should not have a higher training error
      - this is because we can think about solution as a solution by construction:
        
        original layers: copied from a learned shallower model
        
        extra layers: set as identity
        
        => at least the same training error
  - - - H(x) is any desired mapping, hope the small subnet fit H(x)
    - - H(x) is any desired mapping, hope the small subnet fit F(x)
        let H(x) = F(x) + x
      - Directly fit the mapping of these small subnets, actually hoping it will fit another mapping F(x)
      - F(x) - is a kind of residual mapping w.r.t identity
      - the heuristic: if identity were optimal, easy to set weights as 0
      - and: if optimal mapping closer to identity, easier to find small fluctations in addition to the identity
  - - - also introduce the bottleneck design to the residual block
      - so, instead of using 2 or 3 sequential Convs, in each residual block we firt use a 1x1 to reduce the number of channels, and then we do 3x3 Conv, and on the other hand, than we use another 1x1 to increase the dimension back to the original
    - - Deeper ResNets have lower error
      - 34-50-101-152
      - the deepest model (152 layers) have lower time complexity than VGG-16/19, because, if we have better representation power by going deeper, we can use a smaller number of channels => greatly reduce the complexity
    - - Results
        
        256 GPUs
        
        8.192 mini-batch size
        
        ResNet-50
        
        No loss of accuracy
      - key factors
        
        Linear scaling learning rate in minibatch size
        
        Warmup of learning rates in the beginning of training
        
        Implement things correctly in multiple GPUs/machines
  - - - 8-layer LSTM by stacking LSTM blocks with the help of residual connections
      - Usually people are not able to train LSTM with > 4 layers
    - - Residual CNNs on 1-d sequence
      - Key 1: use some delaytions (dilations) which help to capture some long-time dependencies
      - Key 2: stack a lot of such layers - can have very large receptive fields => can have longer dependencies captured
      - When they stack a lot of layers, the residual connections are the key to success
    - - also Residual CNNs on 1-d sequence
  - - - extension to ResNet
      - 3 components to success of GoogleNets:
        shortcuts, bottleneck, multi-branch. Still same 3 key components in all following recent designs
      - Inception: heterogeneous multi-branch (different shape of branches)
      - ResNet: shortcuts, bottlenecck, no multi-branch
      - ResNext: uniform multi-branch (all the branches share the same shape and number of filters)
    - - a simple multi-branch components design in the scenario of ResNet
    - - Concatenation and Addition are interchangable
        
        general propertied for deep NN
        
        not only limited to ResNext
      - Uniform multi-branching can be done by group-conv
      - => can be implemented more efficiently than repeatedly doing many branches
    - - Better accuracy, when manually keeping the same # of FLOPs/params as original ResNets
      - Better trade-off of larger and larger models
      - Can be generalized very well to some other more complicated recognition tasks: Object detection and Instance Segmentation (ex: Mask R-CNN)
      - => Feature still matters! in the current CV community - The success of DL is the success of Deep Features
- - - - One of the most popular
      - Developed under some linear assumption
      - Basically we want some constant factors in the forward and backward pass. And some simple assumption is just that factor for every single layer is 1
      - this initialization is very useful, but was developed under a liner assumption
    - - Initialization under ReLU
      - If the activation is ReLU, we can also have some nice analytical derivation
      - simply modify Xavier case by 1/2
      - this factor is important, because if we have D layers, than a factor of 2 per layer will have exponential effect of 2^D. And this kind of exponential effect is very prevalent in NN, if we don't do things right
    - - required for training VGG-16/19 from scratch
      - deeper (>20) VGG-style nets can be trained w/ MSRA init
        (but deeper plain nets are not better (see ResNets))
      - Recommended for newly initialized fine-tuning
        (e.g. Fast/er RCNN, FCN, etc.)
      - Mathemetical outcome of these 2 types of initialization (1/sqrt(n), 2/sqrt(n)) doesn't directly apply to multi-branch nets (e.g. GoogleNet)
        
        but same derivation methodology is applicable. can do derivations for different types of inception blocks
        
        usually now we don't need to do this, because we have BatchNorm, so now it doesn't matter
- - - - Train mode
        
        mean and std are functions of a batch of x
      - Test mode
        
        mean and std are precomputed* on a training set and will be fixed
        
        precomputed by running average, or post-processing after training
      - Caution!
        
        Research experience: the difference between these 2 modes will create many practical problems in our implementation
        
        We usually have to make sure our BN usage/implementation is correct!