Please enable JavaScript.

Coggle requires JavaScript to display documents.

Training DNNs (automated CL (Weinshall - 2018 - CL by transfer learning…

- - - - ranking - used the confidence of another classifier to estimate the difficulty of training examples
      - scheduling - fixed vs adaptive
- - - - used prior human knowledge to determine difficulty
  - - - approaches to accelerate training/convergence
        
        good weight initialisation (Glorot, 2010)
        
        modified optimisation methods ex Adam (Hinton, 2012) (Bengio, 2013) (Kingma, 2014) - references in Palamuttam (2017)
        
        CL - feed training data to the model in an order which encourages fast convergence
  - - - n-step/stage curriculum - ex in step n are harder than ex in n-1 step
      - curriculum w growing ex - harder ex r added gradually & concatenated w easier ex
    - - regularization in latent var model (SVM)
      - prone to overfitting
    - - regularization
    - - regularization
    - - adversarial MAB
    - - naive CL = n-stage CL
      - mixed strategy = growing CL
      - combined = naive+mix
- - - - MAB/RL methods - use prediction variance, loss values, network complexity etc as reward signals
        
        extrinsic reward method
        
        Fan - ICLR2018
        
        intrinsic reward method
        
        Graves - ICML2017
      - statistical non-uniform sampling/weighting methods (ex: importance sampling)
        
        loss-based, ranking
        
        sampling based on prioritization
        
        Schaul - ICLR2016
        
        sampling based on the distr of network loss value & examples' ranking
        
        Loshchilov - ICLR2016 - Online batch selection for faster training of NNs
        
        importance sampling
        
        sampling based on the distr of prediction variance
        
        Zhao - ICML2015
        
        Alain - ICLR2016
        
        prediction variance
        
        Chang - NIPS2017 - Active-bias-training
        
        sampling based on the distri of examples' class labels
        
        Gopal - ICML2016
      - a separate NN/classifier methods
        
        use prediction variance, loss values etc as input features to estimate examples' weights
        
        Kim - 2018 - ScreenerNet - jointly train both networks
        
        Jiang - 2017 - MentorNet - use a NN as a data regularizer to estimate the weight of each example - separate trained
      - add regularization term to loss fn
        
        Kumar - NIPS2009 - self-paced learning - min-min formulation (i.e. alternative minimization)
        
        Jiang - NIPS2014 - self-paced learning w diversity - min-min formulation
        
        Zhou - ICLR2018 - Minimax CL - min-max formulation using submodular maximization