Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data visualization - Coggle Diagram

- - - - deep knowlge for this tools
      - what I needs
  - - - Effectively use the shapes and features in your figures.
      - Consider the geometries based on objectives like amounts (or comparisons), compositions (or proportions), distributions, or relationships
  - - - A visually appealing design can engage users and increase their interest in the data.
      - Sequential → Progression (low → high)
      - Diverging → Difference (negative ↔ positive)
      - Qualitative → Categories (apple, orange, banana)
- - - - Examples: Bar charts, grouped bar charts, pie charts
    - - Examples: Scatter plots, bubble charts, heatmaps
    - - Examples: Line graphs, area graphs, timelines
    - - Examples: Histograms, box and whisker plots.
- - - - Examples: 1.3, 5.7, 83, 1.5×10⁻²
      - Scale: Continuous
      - Description: Values can take any number within a range — integers, fractions, or decimals.
      - Visualization examples: Line charts, histograms, scatter plots.
    - - Examples: 1, 2, 3, 4
      - Scale: Discrete
      - Description: Values occur in separate units (often integers). Intermediate values don’t exist.
      - Visualization examples: Bar charts, column charts.
  - - - Examples: dog, cat, fish
      - Scale: Discrete
      - Description: Categories without any inherent order. Each is unique.
      - Visualization examples: Bar charts, pie charts.
    - - Examples: good, fair, poor
      - Scale: Discrete
      - Description: Categories have a natural order, but not equal spacing between levels.
      - Visualization examples: Bar charts, stacked bar charts.
  - - - Time flows smoothly; every instant matters
        as Temperature over a day used with Line chart
    - - | Time is divided into fixed intervals | Monthly sales totals | Bar chart
- - - - Data analysis
      - Machine learning
      - Data science
      - AI
    - - Data cleaning
        
        Data cleaning is the process of adding missing data and correcting, repairing, or removing incorrect or irrelevant data from a data set.
        
        identifying and correcting errors or inconsistencies to ensure that the data is accurate and complete.
        
        Data cleaning process include handling missing values, removing duplicates, and correcting incorrect or outlier data.
        
        Data cleaning is essential for effective analysis and improved results.
        
        Tools:
        
        OpenRefine
        
        Trifacta
        
        Tibco Clarity
        
        Ringlead
        
        Talend
        
        Paxata
        
        Cloudingo
        
        Jupyter Notebooks
        
        Steps:
        
        1-Missing Values
        identifying the values that are absent in the dataset and then handling
        
        In data science and machine learning, dealing with missing values is a critical step to ensure accurate and reliable model predictions .
        
        Missing values are common in real-world datasets. Common reasons include:
        
        Data corruption due to poor maintenance.
        
        Human error or incomplete recording.
        
        Intentional omission by users.
        
        Item nonresponse, where participants refuse to answer.
        
        Missing Values Representation
        
        NaN (Not a Number).
        
        NULL or None.
        
        any special characters or number.
        
        Handling of Missing Values
        by these methods
        
        Rows containing the missing values can be deleted
        
        Imputation
        
        Replace missing entries with the average (mean), middle value (median), or most frequent value (mode) of the corresponding column.
        
        KNN Imputation: finds the closest data points (neighbors) based on available features and uses their values to estimate the missing value.
        
        Use statistical method to predict the missing values.
        
        2-Noisy Data
        refers to irrelevant or incorrect data that is difficult for machines to interpret. unclear patterns and lead to inaccurate predictions.
        
        Random Errors: Unpredictable mistakes, like typos or sensor malfunctions, leading to inconsistencies.
        
        Outliers: Data points that are significantly different from others, often due to errors or rare events.
        
        Irrelevant Features: Uninformative attributes that add complexity and risk overfitting.
        
        Inconsistencies: Ambiguous or conflicting data stemming from inconsistent collection methods or coding errors.
        
        Reasons
        
        Data Collection Errors: Faulty instruments, human error during entry, or environmental factors can introduce noise.
        
        Programming Errors: Bugs in software can lead to incorrect data.
        
        Inconsistent Data Entry: Variations in naming conventions, data codes, or input field formats.
        
        Natural Variability: Inherent fluctuations in natural systems can also add noise.
        
        Handling :check::
        
        Binning: Grouping data points into "bins" and then smoothing values by replacing them with bin medians or boundaries to reduce local variations.
        
        Smoothing by bin means: In smoothing by bin means, each value in a bin is replaced by the mean value of the bin.
        
        Smoothing by bin median: In this method each bin value is replaced by its bin median value.
        
        Smoothing by bin boundary: the minimum and maximum values in a given bin are identified as the bin boundaries. Each bin value is then replaced by the closest boundary value.
        
        Smoothing: Techniques like regression or clustering can be used to reduce random noise.
        
        regression
        
        2 more items...
        
        clustering
        
        1 more item...
        
        Outlier Detection: Using statistical methods, such as the interquartile range (IQR), to identify and filter out extreme data points.
        
        Manual Cleaning: Domain-specific adjustments and manual inspection of data can help address specific issues.
        
        3-Removing Duplicates:
      - Data integration
        
        data mapping
        
        data transformation
        
        is also known as ETL (Extract/Transform/Load), which describes the process and use of data transformation.
        
        pre procssing
        
        converting a raw data source into a cleansed, validated, and ready-to-use form.
        
        types
        
        Format Conversion
        
        Data Cleaning ,
        
        📏 3. Normalization / Standardization
        
        ⚖️ 4. Unit Conversion
        
        📊 5. Aggregation
        
        🔗 6. Merging & Splitting
        
        🔢 7. Encoding / Decoding
        
        ✅ 8. Validation Rules
        
        Techniques
        
        🔹 3. Data Generalization
        
        Attribute Construction (Feature Construction)
        
        Data Smoothing
        
        (Equal-width bins) Clustering ,
        Regression (Curve fitting) ,
        Bining,
        Moving Average
        
        🔹 4. Data Aggregation
        
        (totals, averages, counts, groups).
        
        🔹 5. Data Discretization
        
        1- Binning (Equal-width or Equal-frequency bins)
        
        Histogram Analysis
        
        Clustering-based Discretization
        
        Decision Tree Discretization
        
        Entropy-based Discretization
        
        Data Normalization
        
        data profiling (reviewing source data, understanding structure, content and interrelationships, and identifying potential for data projects.)
        
        Structure discovery
        
        Validating that data is consistent and formatted correctly, and performing mathematical checks on the data (e.g. sum, minimum or maximum).
        
        Content discovery
        
        Relationship discovery
        
        data reconciliation
        
        verification phase during a data migration where the target data is compared against original source data to ensure that the migration architecture has transferred the data correctly.
        
        strategy(triple (G, S, M) approach.
        
        G represents the global schema
        
        S represents the heterogeneous source of schema,
        
        M represents the mapping between source and global schema queries.
      - Data reduction
        
        Techniques
        
        Dimensionality Reduction
        
        reducing the number of features in the dataset that are irrelevant or redundant or outdated
        
        methods
        
        4.1.1 Principal Component Analysis (PCA): Projects data into fewer dimensions while retaining maximum variance.
        
        4.1.2 Feature Selection: Keeps only relevant features based on statistical measures or domain knowledge.
        
        4.1.3 Factor Analysis: Groups correlated variables into fewer factors.
        
        Numerosity Reduction
        
        Reduces data volume by using a smaller representation.
        
        methods
        
        4.2.1 Histograms: Group values into bins.
        
        4.2.1.Clustering: Replace data points with cluster centroids.
        
        4.2.3 Sampling: Use a representative subset of the dataset.
        
        4.2.4 Aggregation: Summarize data, e.g., total sales per month instead of daily sales.
        
        Data Compression
        
        encoding mechanisms are used to reduce the data set size.
        
        types
        
        4.3.1 Lossless Compression: Original data can be perfectly reconstructed (e.g., ZIP, run-length encoding).
        
        4.3.2 Lossy Compression: Some data is discarded but main patterns are preserved (e.g., images, audio).
        
        Discretization and Concept Hierarchy
      - Data transformation
      - Data quality assessment
        
        Accuracy: To check whether the data entered is correct or not.
        
        Completeness: To check whether the data is available or not recorded.
        
        Consistency: To check whether the same data is kept in all the places that do or do not match.
        
        Timeliness: The data should be updated correctly.
        
        Believability: The data should be trustable.
        
        Interpretability: The understandability of the data.
  - - - Normalization/Standardization
        
        used to solve the issues in biased may be to happen in future with the data
        
        Most prominent normalization techniques are Min-Max Normalization (scaling to), Z-Score Normalization (scaling to a mean of 0 and std. deviation of 1), and Log Scaling (for skewed data).