Please enable JavaScript.

Coggle requires JavaScript to display documents.

2 .[IR] Doc processing (Document Preprocessing (Approaches (Approaches for…

- - - - Ideally, understand the whole content, and represent the understanding in a computer understandable format
    - - Bag of words representation
        
        Independence between words
        
        Not recognize semantic relationship among words
        
        Index Term
        
        Tokenization: identify the word units for indexing
- - - - decide the basic of a document
      - parsing the document
      - tokenization
      - processing tokens
    - - Synonymy
      - Polysemy
    - - aka WSD problem
      - context is the key
        
        Doesn't work
        
        Bag-of-word techniques already disambiguate
        
        tried to improve precision but incorrect sense assignments would hurt recall
    - - treat multi-word tokens as index terms
      - sources of evidence for identifying phrase
        
        Dictionary lookup
        
        Technical terminology can be very helpful
        
        Linguistic analysis
        
        Syntactic variations
        
        Coordinations
        
        substitutions
        
        Permutations
        
        Statistical analysis
        
        Consider all word bigrams
        
        using pointwise mutual information(PMI)
        
        The words bigrams that give highest PMI values can be considered as phrase
        
        very fast, often used in IR community
      - How are Phrase used?
        
        Precoordinate
        
        Recognize phrase during indexing
        
        replace query constituents with phrase
        
        Postcoordinate
        
        specify phrase at query time
        
        query operator retrieves and merges constituent inverted lists
      - Result?
        
        In large corpus: YES!
        
        High precision effects are desirable
        
        loss of recall is not a problem, why?
        
        but in small corpora it isn't clear
        
        High precision effects are still desirable
        
        loss of recall is more problem