Please enable JavaScript.

Coggle requires JavaScript to display documents.

Prov Chap 10 (Issues from example in reading (It is difficult to predict…

- - - - Text of course has plenty of structure, but it is linguistic structure intended for human consumption, not for computers.
        
        As data, text is relatively dirty. People write ungrammatically, they misspell words, they run words together, they abbreviate unpredictably, and punctuate randomly.
        
        A document is made up of indiciual tokens so basically, a token on a document is one single word.
        
        Bag of words: As the name implies, the
        approach is to treat every document as just a collection of individual words. This approach ignores grammar, word order, sentence structure, and (usually) punctuation.... This style treats every word as a potential key word.
        
        Frequency: The next step up is to use the word count (frequency) in the document instead of just a zero or one.
        
        Many words have been stemmed, suffixes removed.
        
        Stopwords are removed: the and of
        
        Caes are normalized, all lower case
- - - - As presented, the bag-of-words representation treats every individual word as a term, discarding word order entirely. In some cases, word order is important and you want to preserve some information about it in the representation
        
        You can include sequcne of adjacent words if you want things to work better.
        
        N-grams are useful when particular phrases are significant but their component words may not be.
- - - - When looking at results, it is important to look at the predictability. Great curves to look at in regards to predictability are the lift curves.