Please enable JavaScript.

Coggle requires JavaScript to display documents.

NLP - Coggle Diagram

- - - - Vectorization
        
        Every word is uniquely represented by a vector
  - - - Categorizing words in corpus with a part of speech like Noun, Adjective
        
        Can be used as features for machine learning
  - - - Person
      - Location
      - Date
      - Organization
- - - - Tokenization
      - Stemming
      - Lemmatization
      - Part of speech Tagging
      - Used for research & educational purposes
      - Requires more storage space but less memory intensive
      - Supports Natural language generation tasks like translation & chatbots
  - - - Information extraction
      - Natural language understanding
      - Text preprocessing
      - provides pre-trained model & pipelines which can be customized
- - - - Based on past observations
    - - Predefined linguistic rules
  - - - Neural
      - Statistical
      - Rule Based
- - - - Word Cloud :check:
      - -Text Generation :red_cross:
        -Next word Prediction :red_cross:
  - - - Not a alphabetic or numerical character
      - !,#,$,^,@
        Special characters
    - - Commonly used intervening words
        
        Do not add meaning to the context for model input
      - Incorrect removal of stopwords changes meaninig
        "The movie was not good"
    - - Process of breaking raw text into smaller chunks of words or sentences known as tokens
        
        Word Tokenization
        
        Divide text into individual words based on demiliter
        
        Disadvantage- Cannot handle Out-Of-Vocabulary Words
        
        Sentence Tokenization
        
        Break down text into seperate sentences based on dilimiters like fullstop
    - - Lemmatization
        
        Main Idea
        Dictionary based approach
        
        reduce word inflections & inturn reduce redundancy
        
        1) Minimize inflections to their canonical or dictionary form - LEMMA
        2) Before reducing to base form, it also considers the context in which word is useed
        
        Advantages
        
        Considers context in which a word is used
        
        Dictionary based approach. Takes context and returns dictionary definition of word due to which process is slower but higher accuracy than stemming
      - Stemming
        
        Usage of different forms of words
        "connects, connecting,connection"
        
        Affixations (Prefix & suffix)
        
        INFLECTIONS
        
        Stem - Inflection words
        
        -Helps to extract Base word from the given words
        -Stemming Algorithms are rule based
        
        Kinds of Stemmers
        
        Porter
        
        Works on English language
        
        Snowball
        
        -Works on multiple languages
        -Improved version of Porter
        
        Lancaster
        
        More aggressive stemming
        
        Applications
        
        Information Retrieval Systems
        
        Domain Vocabulary Analysis
        
        Text mining
        
        Limitations
        
        Overstemming
        
        If 2 or more words are incorrectly reduced to a single stem, when they should be reduced to 2 or more stem words
        
        If 2 or more words are incorrectly reduced to more than one stem word, when they should be reduced to same stem word
        
        Does not take the context of word, Oonly removes prefix or suffix
        
        Although it is faster than lemmatization, accuracy is low