Please enable JavaScript.

Coggle requires JavaScript to display documents.

Databricks Fundamentals - Coggle Diagram

- - - - Pros:
        
        Flexible data storage
        
        Streaming support
        
        Cost efficient in the cloud
        Support for AI and ML
      - Cons:
        
        No transaction support
        
        Poor data reliability
        
        Slow analysis performance
        
        Data governance concerns
        
        Data warehouses still needed
        
        Implemented complex technology stack environments
        
        Introduced complexity and delay as data teams were stuck in silos
        
        Data duplication
        
        Ai implementation was difficult.
      - The value behind the data was lost.
        Only 32% of companies reported value from data.
- - - - Control plane
        
        Consists of the managed backend services Databricks provide.
        
        These lived in Databricks' own cloud account.
        
        Aligned with the cloud service provider.
        
        Databricks runs the workspace application and manages notebooks, configuration and clusters.
        
        The information is encrypted at-rest and communication to and from control plane is encrypted in-transit
      - Data plane
        
        It's where your data is processed. Unless you choose to use Serverless computing, the compute resources in the data plane run inside the business owner's own cloud account.
        
        If the business decides to host the data plane, Databricks will configure the networking by default.
        The serverless data plane networking infrastructure is managed by Databricks in a Databricks cloud service provider account and shared among customers with additional network boundaries between workspaces and cluster servers.
        
        Clusters are run using the latest hardened system images.
        
        Clusters are typically short-lived, often terminated after a job, and do not persist data after termination
        
        Code is launched in an unprivileged container to maintain system stability. This security design provides protection against persistent attackers and privilege scalation.
      - Support Cases.
        
        Access to the environment is limited to cloud service provider API for automation and support access.
        
        Databricks has a custom-built system, allowing our staff access to fix issues or handle support requests.
        
        Access is limited to a specific group of employees for limited periods and actions are tracked.
        
        For user identity and access, Databricks supports many ways to enable users to access their data:
        
        Table ACL feature, traditional SQL-based statements to manage access to data.
        
        IAM instance profile
        
        Securely stored access key, for external storage
        
        Secrets API, for external resources.
      - Compliance.
        
        SOC 2 Type II
        
        ISO 27001
        
        ISO27017
        
        ISO 27018
        Cloud:
        
        FedRAMP High
        
        HITrust
        
        HiPAA
        
        PCI
        in Databricks platform
        GDPR and CCPA
  - - - Delta Lake is a file-based open-source storage format
        
        ACID transaction guarantees.
        
        Scalable data and metadata handling
        
        Audit history and time travel
        
        Schema enforcement and schema evolution
        
        Support for deletes, updates, and merges.
        
        Unified streaming and batch data processing
    - - The excecution engine has to provide the same performance as a Datawarehouse.
      - Next generation query engine
        
        Cost savings
        Compatible with Spark API with a more general execution framework
        
        Increase spped for use cases such as data ingestiong, ETL, streaming, data science, and interactive quieries directly on your data lake.
      - Performance improvements on
        
        SQL-based jobs
        
        Iot use cases
        
        Data privacy and compliance
        
        Loading data into Delta and Parquet
      - Compatible with Apache Spark Dataframe and SQL APIs
        Coordinates work and resources transparently accelerating portions of SQL and Spark queries without tuning or user intervension