Please enable JavaScript.

Coggle requires JavaScript to display documents.

Deep Learning (Strategy (Human level perf. (problems where algo >HL…

- - - - accuracy and false positives
      - maximise accuracy(O), subject to <=1 false positive(S)
  - - - get labelled data from humans
      - get insight why a person did it right
      - better analysis of bias/var
    - - if T error far from HL, focus on bias
      - if T error close to HL, focus on variance
      - it is the difference between training error and HL, always reduce this first
    - - online advertising
      - recommendations
      - logistics
      - loan approvals
      - all these learn from structured data, not natural perception tasks, teams have access to far more data than a human can look at
      - fringe: speech recog, image recog, medical imaging analysis
    - - for avoidable bias
        
        bigger model
        
        train longer, better optim
        
        hyperparam search
        
        diff. nn architecture
      - for variance
        
        more data
        
        more regularisation
        
        dropout
        
        augmented data
        
        hyperparam search
  - - - understand diff between T and D sets- manual analysis
      - make them similar, collect more data like D/Te
      - syntesize data: add noise, risk of overfitting to noise, shouldn't simulate data from a small subset of all possible types
- - - - as it is computed on a minibatch, the mu and var is noisy, mult. and additive noise(dropout has mult. noise), by adding noise it prevents later hidden units from depending on any one hidden unit
      - reg effect is slight as the noise is small
    - - no mini batch available
      - estimate mu and var using ewa across training mini-batches
- - - - l2 reg = lambda/2m * l2norm(w)
      - not done for bias
      - with L1 reg W becomes sparse, compresses model
      - Frobenius norm, l2norm of all w of all layers
      - for GD with this
        
        earlier: wl = wl - lr*dwl
        
        now: dwl = (from backprop) + lambda/mwl, so wl = wl - lrlambda/m wl - lr(from backprop)
        
        weight decay (1 - lrlambda/m)w - lr*(from backprop)
      - L2 Reg - why it works?
        
        shrinking the Fnorm prevents overfitting
        
        if Lambda is high, pushes w to 0, zeroes out/minimizes impact of hidden units, makes nn simpler, pushes to high B case
        
        every linear activ. will b roughly linear, forces small weights which keeps act. in the linear region(tanh case)
        
        L2-regularization relies on the assumption that a model with small weights is simpler than a model with large weights. Thus, by penalizing the square values of the weights in the cost function you drive all the weights to smaller values. It becomes too costly for the cost to have large weights! This leads to a smoother model in which the output changes more slowly as the input changes.
      - prevents overfitting/reduces V
  - - - create mask for choosing neurons in act
      - elementwise multiply with act
      - scale up act by keeping prob, a3 /=keep_prob
      - done because, shutting off act neurons, reduces expected value of z from the act., hence scale to preserve expected value
      - at test time, this technique makes easier , prevents scaling problem as no DR at test time, no need to add scaling param
    - - a neuron should not rely on only few features, should spread out weights across input features, should not memorize data set by recording strong presence of few features
      - this too shrinks weights like l2reg and prevents overfitting, it's an adaptive form of l2reg
    - - for layers with dense connections, higher risk of overfitting, keep_prob can be low
      - for other layers can be high, even to 1.0 for layers with few neurons where we don't care about of.
  - - - plot dev set error
      - stop when dev error starts going up
      - works coz w keeps going bigger, stopping at midsize w, picks an nn with smaller norm
- - - - larger the #of act. coming into a neuron, smaller should be the initial weights
      - one way is to set the variance of the w to be 1/n_l-1, for relu 2/n_l-1; i.e. w = w*sqrt(2/n_l-1), sampling a random dist. with this var
  - - - check for which components grad_approx and grad are far off
      - could be bad computation of the concerned component gradients
      - remember to add regularisation term to gradient computation
      - doesn't work with dropout
      - w,b may become larger, gd might be inaccurate, run grad check after evry few iterations
  - - - cost doesn't go down smoothly
      - batchsize=m -> BatchGD; batchsize=1 ->StochasticGD
      - SGD: loss of speed from vectorization, random walk to minimum
      - BGD, slow, steady strides to minimum, use for small training set
      - MBsize could be a hyperparameter, typically 2^{6-10}
    - - param beta<1, gives weight to the previous 1/(1-beta) instances and the rest to the current instance, large beta makes curve smooth but slow to adapt to changes, small beta puts weight to less #pastinstances and jagged curve, fast to g=detect changes
      - attaches an exponentially decaying weight to all previous instances, lower beta makes the exp func drop quickly hence puts weight to recent past, large beta makes expfunc drop slowly, hence putting weights to further back
      - bias correction: use v_t/(1-beta^t), this prevents the initial avgs from being too small, also does not affect latter avgs as the deno term goes to 0 with higher t
    - - calc EWA of grad
      - with gd forced to use slow lr to prevent overshooting
      - we want shorter strides across, but larger strides towards the min
      - update w&b with vdw and vdb(EWAs)
      - smooths out the steps in gd, the avg in vertical directions cancel out to close to 0 but in the hori direction still remains large as all of them are pointing towards the min
      - if it is a bowl shaped contour, the dw term is accel, the vdw of previous iterations is velocity and beta is friction, gd gains momentum as you climb down
      - vdw0 = vdb0 = 0
      - can omit the 1-beta scaling from the dw update in calculating the avg, then can scale lr with 1/1-beta
    - - slows down movement in vert and accels movement in hori dir
      - take ewa of squares of grads
      - update with dw scaled by 1/sqrt(s_dw)
      - so if grad is large, then update is divided by large number
      - add eps before sqrt to prevent division by very small #
      - allows using larger lr to speed up training
    - - puts momentum and rmsprop together
      - compute bias corrected vdw,sdw, vdb, sdb
      - more weight to the squared grad
    - - near the minimum, optim might keep wandering
      - lr = (0.95)^t * lr_0
      - lr = k/sqrt(t)* lr_0
      - 1/2 every n steps, discrete staircase
      - manual decay