Please enable JavaScript.

Coggle requires JavaScript to display documents.

NLP and word embeddings (Word embeddings (Can help with analogy reasoning,…

- - - - Let's say
        I want a glass or orange juice to go along with my cereal
        context: orange
        target: one of [juice, glass, my]
      - Model
        
        Vocab size = 10 000 000
        x ------------------------> y
        context c "orange" -> target t "juice"
        oc -> E -> ec -> softmax -> y^<formulas for model>
        
        Problems with softmax classification
        computational volume – you have to go through 10k words in vocabulary
        solution for that could be hierarchical softmax (via tree)
    - - I want a glass or orange juice to go along with my cereal
        select positive and randomly other examples
        context, word, target
        orange, juice, 1
        orange, king 0,
        orange, book, 0
        orange , the, 0
        orange, of, 0
        k = number of other examples = 4
        usually k = 5..25
      - Then we train the model, using softmax (?)
        we don't train 10k softmax,
        but instead 10k binary classification problems
        which turns out much less computationally expensive
      - How to select negative examples?