Please enable JavaScript.

Coggle requires JavaScript to display documents.

Optimize Our Model (:pineapple:complexity of models (:one:Underfitting (…

- - - - :zap:The ability of description of a simpler model is far away from the real label; It is difficult for the complex model to find the optimal state
      - :arrow_right:How do we determine our model's bias & variance
        
        :one:If our model cannot even fit the training data, we have large bias
        
        :arrow_right:underfitting
        
        :two:If our model can fit the training data but has large error on testing data, then we probably have large variance
        
        :arrow_right:overfitting
  - - - :two:regularization
      - :one:more data
      - :three:dropout
        
        :explode:training时去除\(p\%\)的神经元,testing的时候使用全部的神经元,并且权重乘上\(1-p\%\)
        
        :poultry_leg:为什么训练时权重要乘上\(1-p\%\)呢
        
        :palm_tree:直观上理解testing使用全部神经元结果会更大,所以需要缩小权重
        
        :icecream:linear neuron来理解
        
        :poultry_leg:那么如果使用sigmoid作为activation function的话,那么不就不满足\( \dfrac{1}{2} \)吗,这样为什么还可以使用
        
        :maple_leaf:很神奇的是,确实可以使用
        
        :arrow_right:使用linear activation function配合dropout效果最好
        
        :right
        
        :key:ensemble
        
        :palm_tree:每次dropout相当于都是新networks,而最后使用所有的新networks,并且通过权重乘上\( 1-p\% \)来作为average
        
        :explode:训练多个networks,最后使用所有的networks,然后取average
        
        :bulb:形象的解释
        
        :one:给模型绑上重物,然后训练时卸下来
        
        :two:每个神经元不能偷懒,必须都起作用
    - - :arrow_right:DL需要关注training set上的准确率,来判断是否训练好
- - - - :one:we divide training set into training set & validation set.
      - :two:We train our model on new training set
      - :three:We select the model with lowest error on validation set
      - :four:We can then train the selected model on the original training set (if we are worried that the training data is less)
      - :five:We can put the selected model on the public testing set which usually generates the same error as that of validation set
- - - - :maple_leaf:Adagrad
- - - - :musical_note:good sides
        
        :one:不使用Feature Scaling的话必须使用Adagrad,保证不同权重的平衡性
        
        :two:保证参数更新都是准确地往最低点走.(不使用Feature Scaling的话,椭圆形地沿着法线走)
      - :snowflake:ideas
        
        :one:正态标准化(Z-score方法)
        
        :two:极差变换法
        
        :three:线性比例变换法
      - :palm_tree::原因
        
        :one:如果特征值1一直是0,1,2;而特征值2一直是100,200;那么显然w2的变化对loss带来的影响大于w1
        
        :two:加快收敛速度
        
        :bread:不feature scaling的话,loss function就是椭圆形,每次沿着垂直方向走却不是离极小值最近的方向
      - :explode:将若干样本的某个特征值做正态标准化,从而保证每个特征值符合(0,1)正态分布