Please enable JavaScript.

Coggle requires JavaScript to display documents.

Sequence to sequence architectures (Speech recognition (CTC cost for…

- - - - Length normalization
        Beam search is: maximizing this probability
        arg max(y) П (from 1 to Ty) P(y<t> | x, y<1>, ... y<t-1>)
        and these probabilities are usually less than 1
        so in practice we can do
        arg max(y) SUM (from 1 to Ty) log P(y<t> | x, y<1>, ... y<t-1>)
        downside of it is that it prefers shorter translations due to fade of product. Instead we could do is use
        (1 / Ty) * SUM (from 1 to Ty)...
        or a bit softer approach
        (1 / (Ty^alpha))...
        where alpha in [0..1] (1 - complete normalization, 0 - no normalization)
      - How to chose B?
        Higher B – more considered possibilities,
        but takes longer
        Smaller B – worse results, faster
        In practice it's around ~10 in production
        In research it's up to 3000
        When it's very large, quality return is diminishing
        Beam Search runs much faster than Breadth first search/Depth first search. But Beam does not guarantee to use maximum arg
    - - we take
        Human version: Jane visits Africa in September (y*)
        Algorithm version: Jane visited Africa in September (y^)
        
        Case 1:
        P(y*|x) > P(y^|x)
        Beam search chose y^, but y* attains higher P(y|x)
        Conclusion: Beam search is at fault!
        
        Case 2:
        P(y*|x) <= P(y^|x)
        y* is better translation than y^
        But RNN predicted P(y*|x) < P(y^ | x)
        Conclusion: RNN is at fault!
      - Process
        Human | Algorithm | P(y*|x) | P(y^|x) | At fault?
        Jane visits | Jane visited | 0.002 | 0.001 | B
        ... | RNN
        ...
        figure out which component is at fault for most errors. If Beam, then maybe increase B, otherwise add dropout and so on