Please enable JavaScript.

Coggle requires JavaScript to display documents.

RNN (:sweat_drops:gradient vanishing & gradient explode (:one:\( h_t …

- - - - :arrow_right:链式规则导致W累乘
        
        :arrow_right:通过SVD最后发现如果W的奇异值大于1,就会因为累乘而梯度爆炸,相反就会消失
- - - - 抽象的RNN循环体结构
      - 简单RNN的详细计算过程
    - - import tensorflow as tf lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(lstm_hidden_size) state = lstm_cell.zero_state(batch_size,tf.float32) loss = 0.0 for i in time_steps: if i > 0: tf.get_variable_scope().reuse_variables() lstm_output, state = lstm_cell(curr_input, state) final_output = nn(lstm_output) loss += calc_loss(final_output, expected_output)
  - - - :star:details
      - :poultry_leg:为什么使用tanh?
        
        :one:使用ReLU的话,W可能线性相乘,如果W>1就会爆炸
  - - - :palm_tree:存储的信息更有实际含义
- - - - 函数
    - - 图像描述
        
        NeuralTalk2
    - - 举例
        
        视频输出文本
        
        C3D: Generic Features for Video Analysis
        
        分类文本
        
        文本是许多词,所以many
    - - 举例
        
        语音翻译
        
        视频描述
- - - - :warning:dba
        
        :explode:ba的导数是该batch的和,也就是说ba的shape是(rnn_size, 1),而\((1-tanhx^2)\)的shape和a一样,是(rnn_size, batch_size).所以直接在axis=1上做sum并keepdims
    - - :link:bptt formula + lstm coding step by step
      - :star:在计算\( O_t \)的error时,只用计算\(h_t\)过来的error; 而在计算用来derive \( c_{t} \)的时候,不仅考虑\( h_t \)过来的error,还需要\( c_{t+1} \)的误差; 所以在图中第二行需要加上\( \delta state_{t+1} \odot f_{t+1} \)
      - :icecream:examples
        
        :one:error of forget gate + :two: error of \( W_o \) + :three: error of \( a_{prev} \) + :four: error of \(x\)
- - - - :fire:上下句子间有上下文联系
        
        :arrow_right:下一个句子直接放到上一个句子后面
      - :fire:小
        
        :check:全部载入内存
    - - :star:log perplexity和交叉熵是等价的.使用sparse_softmax_xxx从[-1,vocab_size]和真正labels的[-1]计算出来的loss就是log perplexity.最后只要再np.exp就行了
      - :maple_leaf:使用sparse_softmax来获得当前batch_size * num_steps个单词的log perplexity,做平均值,得到要优化的loss.然后我们迭代所有的数据集的时候,不断累加这个loss,最终将loss除以num_batches就可以了
      - :star:针对的是整个数据集,但最终结果要平均到每个单词
  - - - :apple:WMT数据集
        
        :fire:上下句子间无关系
        
        :arrow_right:通过padding来填充,保证第二个维度相同.再通过dynamic_rnn的sequence_length来跳过填充的部分
        
        :red_cross:直接组成1个大的字符串
        
        :fire:大
        
        :arrow_right:不能一次性载入内存,所以使用TensorFlow的Dataset
      - :checkered_flag:翻译文本
    - - :snowflake:Encoder用CNN提取图像的特征,输出给Decoder,Decoder负责生成Caption
      - :question:image feature如何给Decoder(LSTM)使用?
        
        :one:貌似有直接concat的方法?
        
        :two:image feature通过LSTM一个时间步得到cell state,然后利用这个cell state初始化LSTM,此时的input就是<sos>
        
        :warning:notice
        
        :one:image feature size == embedding_size
        
        :dagger_knife:转换的方式可以直接使用全连接层
        
        :bread:image feature的size必须转换到embedding_size,因为LSTM有个参数的shape是\(( \text{lstm_size}, \text{input_size} )\),所以如果image feature的size和sentence的size不一样的话,就不能共用LSTM了
      - :reminder_ribbon:beam search
        
        :explode:When using the decoder output, the first timestep will have vocab_size probability, and the first k with the highest probability will be used to input to the next layer. For the next layer, we also select the first k with the highest probability which is calculated by multiplying the probability of previous words by the probability of current word (ie, the probability of the current entire sentence)
        
        :link:introduction to beam search
    - - :one:attention
      - :two:加入高层语义
        
        :bread:简单来说,就是在图片中提取出该图片包含哪些物体(多标签分类任务)