Please enable JavaScript.
Coggle requires JavaScript to display documents.
Training DNN Aurelien Geron Book - Coggle Diagram
Training DNN Aurelien Geron Book
Problems
Vanishing gradients/Exploding gradients
Solutions
Glorot and He Initialization
Nonsaturatng Activation Functions
ReLU (rectified linear unit)
Leacky ReLU
ELU (exponential linear unit)
SELU (scaled)
Batch Normalization
momentumm hyperparameter
axis hyperparameter
Gradient Clipping
Lack of data (training data)
Solutions
Reusing pretrained Layers = Transfer Learning
Unsupervised Pretarining
Pretraining on an Auxiliary Task
Self-supervised learning
Slow training
Applying a good initialisation strategy for the connection weights
good activation functions
Batch normalization
faster optimizers
Learning rate scheduling
There are many different strategies to reduce the learning rate during training.
Overfitting
Regularisation
Early stopping
Batch normalization
l1 and l2 Regularization
Dropout
Dropout rate 'p'
Max-norm Regularization
Normalize the input features