Please enable JavaScript.

Coggle requires JavaScript to display documents.

IBM_DE - Coggle Diagram

- - - - Structured
        
        OLTP systems, databases,
      - Semi-structured
        
        emails, XML, TCP/IP packets,zip files
      - Unstructured
        
        audio, video, images, pdf, webpages, social media feeds
    - - Entering a value greater than the defined length of a data field results in loss of information
    - - Different types
        
        Document-based
        
        preferred for e-commerce platforms, medical records storage, CRM platforms, analytics platform
        
        MongoDB, DocumentDB, CouchDB, Cloudant
        
        :red_cross: complex search queries & multi-operation transactions
        
        Column-based
        
        Data is stored in cells grouped as columns of data
        
        logical groupings of columns = column family
        
        all cells corresponding to a column are saved as continuous disk => making access and search easier and faster
        
        Suitable for (1) systems that require heavy write requests
        (2) storing time-series data
        (3) weather data (4) IoT data
        
        Cassandra, HBase
        
        Graph-based
        
        Use graphical model to represent and store data
        
        Useful for finding connections between different pieces of data
        
        (1) Social networks
        (2) product recommendations
        (3) network diagrams
        (4) fraud detection
        (5) access management
        
        :red_cross: large volumes of transactions & analytics queries
        
        Neo4J, CosmosDB
        
        Key-value
        
        Redis, Memcached, DynamoDB
      - Advantages
        
        Ability to run as distributed system scaled across multiple data centers
        
        efficient and cost-effective scale-out architecture with addition of new nodes
    - - Data warehouses
        
        Benefits of cloud-based
        
        lower costs
        
        Limitless storage; compute capabilities
        
        Scale on pay-as-you-go basis
        
        Faster disaster recovery
        
        teradata, oracle, IBM DB2, Amazon redshift, Google BigQuery, Cloudera, Snowflake
      - Data Marts
        
        sub-section of data warehouse, built for particular business function, purpose or community of users
        
        3 types
        
        Dependent
        
        offer analytical capabilities for a restricted area of data warehouse
        
        Source data is already cleaned and transformed
        
        Independent
        
        created from external data
        
        need to carry out transformation process on the source data
        
        Hybrid
      - Data Lakes
        
        store large amounts of structured, semi-structured, unstructured data in their native format
        
        exist as a repository of raw data straight from the source, to be transformed based on the use case
    - - Velocity, Volume, Variety, Veracity, Value
      - Tools
        
        Hive
        
        open-source data warehouse software for managing large dataset files in HDFS or HBase
        
        High latency
        :red_cross: Not suitable for apps with fast response time
        
        Read-based
        :red_cross: Not suitable for OLTP
  - - - Data Ingestion | Collection
        
        connect to the source systems => bring data to data platform in streaming | batch modes
        
        maintain information about data collected in the metadata repository
        
        Data Flow, IBM streams, IBM streaming analytics, Amazon Kinesis, Kafka
      - Data Storage & Integration
        
        transform and merge extracted data, either logically or physically
      - Data Processing
        
        read data in batch | streaming from storage => apply transformations
        
        Structuring (e.g. joins, unions, etc); Normalization; Denormalization; Data cleaning
        
        storage & processing can occur in the same layer (e.g. RDBMS)
      - Analysis & User Interface
        
        delivers processed data to data consumers
      - Data Pipeline