Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data Quality Problems, Data Preprocessing Techniques - Coggle Diagram

- - - - Binning
        
        Sort and Distribute Data into a number of equal-sized buckets or bins.
        
        Smoothing by bin: each value in a bin is the mean of the bin.
        
        Smoothing by bin boundaries: min-max values of bin.
        then each bin value is then replaced by the closest boundary value.
        
        Lecture Example: Slide 18
      - Clustering
        
        Values outside the set of clusters are outliers.
  - - - Reasons for Redundancy: Entity Identification and Derived/ Derivable Attributes
      - Can be detected using correlation analysis and covariance analysis
      - Correlation Analysis
        
        Correlation Coefficient (Numerical)
        
        Chi-square (Categorical)
- - - - relevant subset attributes
        
        remove redundant or irrelevant attributes.
      - Heuristic Methods
        
        Forward Selection
        Initial Set {}
        Then select until you get relevant attributes.
        
        Backward Elimination
        Initial Set {A1, ..., An}
        Then eliminate until you get relevant attributes
  - - - Regression
    - - Histograms
      - Clustering
        
        Represents data as centroid.
      - Sampling
        
        Represents data by a smaller, random sample.
        
        Effective sampling if:
        
        the sample represents approximately the interest of the original data.
        
        The sample works almost as the entire data set.
        
        Types of Sampling
        
        Simple Random Sampling
        
        With Replacement (The object can be picked up more than once)
        
        Without Replacement (The object is picked up only once)
        
        Stratified Sampling
        
        Split data into classes/strata then randomize and pick.