Please enable JavaScript.

Coggle requires JavaScript to display documents.

Databricks - Coggle Diagram

- - - - Apache Spark
      - DBIO
        
        caching
        
        Disk Read/Write
        
        File Decoding
      - Serverless
        
        Shared Pool
        
        No Manual Config
        
        Cluster utilization
      - ML Frameworks
        
        Tensorflow
        
        PyTorch
        
        Keras
        
        GraphFrames
        
        Third Party Libraries like Scikit-learn, DataRobot etc.,
      - Delta Lake
        
        open source storage layer on Data lakes
        
        ACID Transaction Support
        
        Schema Enforcement
        
        Full DML Support
        
        Time Travel
      - consists of
        
        Specific apache spark version
        
        Set of optimizations
        
        Ubuntu OS and its System Libraries
        
        Java, Scala, Python, R libraries
        
        ML Libraries
        
        GPU Libraries
    - - Azure or AWS
      - Provision Cluster VMs with Single Click
      - Setup Multiple Clusters
      - Preinstalled with DB Runtime
      - Databricks File System
        
        Abstraction Layer
        
        Cloud Storage to persist the data
    - - Interactive
        
        visualize the data using charts
        
        Collaborate in real time and Track Changes
        
        Build interactive dashboards
        
        Explore data like SQL IDE
      - Production
        
        Build workflows
        
        Deploy Spark jobs
        
        Monitor jobs
  - - - Types
        
        Interactive
        
        Job Cluster
      - Nodes
        
        Worker Node for Data processing
        
        Driver Node for Co-ordinating and distribution of tasks to worker Node
      - Setup
        
        Choose DB Runtime Version
        
        Node Configuration
        
        Specify autoscale limits
        
        auto termination
    - - organize assets in folder
      - Define access control on objects
      - Setup Version Control
        
        GitHub
        
        BitBucket
        
        Azure Devops
    - - Code in any spark supported Language
      - Workflows - invoke notebook with parameters
      - Execution
      - Built in Visualization - turn data into charts or dashboards
      - supports collaboration
    - - Execution of notebook or JAR
      - run immediately or on schedule
      - Create job clusters to run jobs
      - Different Cluster configuration
      - Monitor job runs and setup alerts
    - - Install 3rd party libraries supported by Spark
      - Import library into notebook to work
      - Scope
        
        Cluster
        
        Notebook
    - - structured datasets
      - Equivalent of dataframe
      - Uses underlying cloud storage
      - Directly query or write to tables
- - - - Batch
      - Streaming
      - Machine Learning
      - Advanced Analytics
    - - Infrastructure Management
      - Manual Configuration
      - Upgrade Challenges
      - Tooling & Integration Complexity
      - Lack of User Interface
      - Difficult to Collaborate on Projects
- - - - Partitioned
        
        Partitions are stored in the memory
        
        dataset is split into Partitions
        
        Nodes process data in Parallel
      - Native Language Support
        
        SQL
        
        Scala
        
        Python
        
        R
        
        Java
      - In-Memory
      - Read-only
        
        Can be converted into another RDD
        
        applies either transformations or actions
        
        Action Generates result
      - Resilient
        
        Track Creation using Lineage Graph
        
        Reconstructed in event of a failure
        
        Provides fault tolerance
    - - Structured Processing
      - Querying
      - Common Library
    - - Built on top of RDD
      - Data is organized into named Columns
      - Imposes a structure on the data
      - Spark Engine can apply Optimizations
      - Allows for structured data processing