Please enable JavaScript.

Coggle requires JavaScript to display documents.

Deep Learning ((W.10)FFNN (Network structure (Activation function is…

- - - - If known the data range, use min-max
  - - - Usually between 5 & 30
    - - Usually < 1000, max few 100 neurons per hidden layer
    - - Use fat short
    - - Sigmoid
        
        0 to 1
      - Tanh
        
        -1 to 1
      - Relu
        
        0 to X
    - - Cross Entropy
        
        *面試題: Why use Cross Entropy? (W10理論, p.46)
      - Squared Error
- - - - One-hot encoding of the original data -> 5 grams -> 將原有的data拉直成一個vector -> 中間的字為label,前後兩個字各為features -> train a neural network of one layer
        
        Why 5 grams? Too long will cause semantic meaning shifting; Too short will lose the context
    - - The middle word is a feature,前後兩個字為labels
- - - - Input gate: 決定input是否重要
      - forget gate: 決定是否要記憶上一次input的memory
      - memory cell: 決定是否Update這次的input在上一次的memory cell裡
        
        Memory cell在update完之後被當成下一次的input
      - output gate: 要不要將最後的值產出?
    - - Usually between 32 and 256. The largest so far is 2048. 並可以搭配金字塔結構,一開始memory dimension小一點
    - - 當regularized控制的範圍過大,並加上先前的memory cell一直記憶著 (把很大的值一直傳下去)
      - How to prevent: 強迫weights keep在lambda以下(regularization)
- - - - PCA & SVD are linear compression