Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data Wrangling (Data Pre-processing (1.2 Data Quality Assessment…

- - - - Segmentation: Parsing data requires segmenting values into separate elements
        
        Rule-based;
        
        Pattern matching languages;
        
        Probabilistic methods;
  - - - Binning
      - Regression
      - Clustering
      - Manual Inspection of possible outliers
  - - - Step-wise backward elimination
      - Decision-tree induction
      - Step-wise forward selelction
  - - - Linear regression: fit the data to a straight line.
      - Multiple regression: to transform to non-linear functions;
      - Log-linear models
    - - Histograms
        
        Binning Method
      - Clustering
      - Clustering Techniques
      - Sampling
- - - - Threshold Based Classification
      - Probabilistic Classification
      - Cost Based Classification
      - Rule Based Classification
      - Machine Learning Based Classification
    - - Phonetic encoding
      - Soundex Algorithm
      - Sorted Neighbourhood
      - Canopy Clustering
      - Q-gram Based Blocking
    - - Linkage Quality
        
        Accuracy
        
        Precision
        
        Recall
        
        F-measure
      - Linkage Complexity
        
        Reduction Ratio
        
        Pairs Quality
        
        Pair Quality
      - Clerical Review
    - - Q-gram Based String Comparison
      - Edit Distance
      - Bag Distance
      - Jaro-Winkler String Comparison
      - Statistical linkage key SLK-581
  - - - Complement Union
      - Minimum Union
      - Full Disjunction
- - - - Secure hash encoding: Use a one-way hash-encoding function (like SHA) to encode value and then compare the hash-encoded values to identify matching records
      - Noise and differential privacy: Add noise to overcome frequency attack at the cost of loss in linkage quality;
      - Generalisation techniques: Generalises the records to overcome frequency attacks
      - Encryption and SMC
        
        Commutative and homomorphic encryptions are used
      - Bloom filters: Probabilistic data structure