Please enable JavaScript.

Coggle requires JavaScript to display documents.

Cloud-Data-Platforms - Coggle Diagram

- - - - semistructured data
        
        JSON/XML
        
        SQL is a great query language, but it’s not a great programming language because it lacks many of the tools today’s software developers take for granted: testing, abstractions, packaging, libraries for common logic, and so on
  - - - allows you to store data on AWS S3
      - process the data on an AWS EMR cluster without permanently storing any data on AWS EMR machines
  - - - shortcomings
        
        Adding hardware to scale the system often takes months
        
        Storage and compute are not separate, meaning that while the same hardware can be used for both storage and compute
        
        From a developer perspective, its use of an “open” toolset makes it very flexible, but its lack of cohesiveness makes it challenging to use
        
        It isn’t easy for users who want to access the data. Its unstructured approach to storage,
        
        It is a complex system with many integrated components that run on hardware in a data center
      - Velocity
        
        to do real-time processing on Hadoop using products such as Hive or MapReduce or, more recently, Spark.
      - Volume
        
        took advantage of distributed processing and storage across less expensive commodity hardware that could be added in smaller increments as needed.
      - Variety
        
        processing in Hadoop could be done on any data type
  - - - more expensive and complicated to store and process these growing volumes and velocities of data in a data warehouse
      - struggles to cost-effectively accommodate the variety of data that IT must make available
      - instead of in a combination of a data lake and a data warehouse
    - - easily and cost-effectively handles an almost unlimited VVV
      - not usually organized in a way that’s useful to most users—business users in particular
      - Much of the data in a data lake is also ungoverned
      - a data lake is almost always coupled with a data warehouse
        
        data warehouse serves as the primary governed data consumption point for business users
        
        direct user access to the largely ungoverned data in a data lake is typically reserved for data exploration either by advanced users, such as data scientists, or other systems
    - - data warehouse
        
        associated ETL tools
      - data lake
        
        occur in the data lake itself
        
        new forms of processing, such as streaming
    - - new processing technologies available in the cloud
      - a cloud data warehouse
      - a cloud data lake
  - - - store
        
        process
        
        serving
        
        cloud data warehouse
        
        API
        
        Data exports