Please enable JavaScript.

Coggle requires JavaScript to display documents.

Big Data - Coggle Diagram

- - - - gigabytes to terabytes
    - - petabytes to zettabytes
  - - - centralized
    - - Distributed
  - - - structured data (tables, rows, and columns)
    - - unstructured (text, videos, images) and semi-structured (JSON, XML, logs) data.
  - - - commodity hardware (cheap, simple servers) connected in large clusters.
    - - high-end, expensive servers
  - - - Typically write once and read many times (e.g., logs, clickstream data, sensor data).
    - - Allows frequent reads and writes (e.g., bank transactions).
  - - - Supports a dynamic schema (flexible, can handle different structures without strict rules).
    - - Uses a static schema (structure must be defined before inserting data).
  - - - best for massive, varied, fast-growing datasets from many sources.
    - - best for structured, smaller-scale, transactional data.
- - - - 1 Storage: Where do we store these enormous amounts of data? Solutions include distributed file systems like HDFSor cloud storage.
      - 2 Access: How can we retrieve and access such large datasets quickly and efficiently? This requires advanced search and retrieval systems.
      - 3 Processing: How can we analyze and process petabytes or even zettabytes of data? Technologies like MapReduce and Apache Spark are widely used.
  - - - 🔹Real-Time Processing (Fast – like a cheetah)
        
        Data is captured instantly as it streams.
        
        Fed directly to machines/systems.
        
        Processed immediately.
        
        Quick actions are taken (e.g., fraud detection in banking, live traffic updates in Google Maps).
      - 🔹 Batch Processing (Slow – like a snail)
        
        Data is collected first.
        
        Cleaned and stored in chunks.
        
        Processed later in batches.
        
        There is a waiting time before action is taken.
        
        Example: Payroll systems, where employee data is processed monthly.
- - - - Example: Market research reports, customer demographic data, and sales performance metrics.
- - - - Cleaning (removing errors/inconsistencies)
      - Integration (combining data from different sources)
      - Reduction (minimizing data volume while preserving information)
      - Transformation (converting data into a suitable analytical form)
        
        Smoothing
        
        Binning
        
        Clustering
        
        Regression
        
        Aggregation
        
        Generalization
        
        Discretization
    - - MapReduce Tasks: Handles distributed processing across multiple nodes.
      - Stream Computing: Processes real-time data streams.
      - Database Analytics: Performs in-depth data analysis using stored data.
      - type:
        
        • Descriptive Analytics:
        
        • Predictive Analytics:
        
        • Prescriptive Analytics:
  - - - Data Visualization: Presents results in graphical formats.
      - Real-Time Monitoring: Tracks ongoing operations dynamically.
      - Decision Support: Provides analytics-driven recommendations and intelligence.
- - - - Feedback, reviews, and service requests from customer support
- - - - Hadoop Distributed File System (HDFS):
      - Cloud Storage:
      - NoSQL databases:
  - - - handle missing values,
      - correct errors
      - Remove duplicates,
    - - Convert data into a suitable format for analysis.
    - - tools
        
        Apache Hadoop
        
        Apache Spark
        
        Apache Hive
        
        H2O
        
        Azure Machine Learning (Azure ML)
  - - - • Tableau:
        •
      - Qlik Sense: