Please enable JavaScript.

Coggle requires JavaScript to display documents.

DataBricks - Coggle Diagram

- - - - Control who has access to which data
    - - Capture and record all access to data
    - - Capture upstream sources and downstream
        consumers
    - - Ability to search for and discover authorized assets
  - - - Data Owner
      - Metastore Admin
      - Account Admin
      - Identity Admin
      - Cloud Admin
      - Workspace Admin
    - - User
      - Account Administrator
      - Service Principal
      - Service Principal with administrative privileges
      - Groups
  - - - Share tables from Region A with Region B
    - - Account-level Identities
        
        Manage all identities at the
        account-level
      - Groups
        
        Use groups rather than users to assign access and ownership to securable objects
      - Service Principals
        
        Use service principals to run
        production jobs
- - - - DLT Syntax
        
        Define DLT
        
        CREATE OR REPLACE [STREAMING] LIVE TABLE
        
        quality constrains
        
        ON VIOLATION
        
        FAIL UPDATE
        
        DROP ROW
        
        OMITTED
        
        Comment
        
        table properties
        
        Processing data - Pipeline
        
        Step 1: Streaming ingestion with AutoLoader
        
        FROM cloud_files()
        
        spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "json")
        
        bronze_layer
        
        Step 2:Quality Enforcement
        
        Quality Constrains
        
        bronze_cleaned
        
        Step 3: Processing CDC Data with apply_changes
        
        APPLY CHANGES INTO
        
        dlt.apply_changes()
        
        sliver_layer
        
        Step 4: Live views
        
        Troubleshooting DLT Syntax
        
        SQL
        
        missing LIVE key work - create clause
        
        missing STREAM() - FROM clause
        
        missing LIVE.table_name key word - FROM clause
        
        Python
        
        missing dlt.table() decorator
        
        agrument
        
        dlt.read() table from DLT pipeline <> NOT spark.read()
    - - Config
        
        Pipeline mode
        
        Triggered
        
        Continuous
        
        Notebook libraries
        
        Storage location
        
        If not specified, DLT will automatically generate a directory
        
        store logs, tables, and other information
        
        not specified -> tables will not be registered to a metastore,
        but will still be available in the DBFS
        
        Target
        
        Cluster mode, Min Workers, Max Workers
        
        source
    - - Development vs Production Mode
  - - - Bronze layer
      - Silver Layer
      - Gold layer
  - - - Usages
        
        1, Ingest files from cloud storage
        
        FROM cloud_files()
        
        Stream data from Delta table
        
        FROM STREAM(table)
- - - - Unity Catalog
        
        Metastore
        
        Access Control
        
        Data Lineage/ Explorer
      - Web App
      - Cluster Manager
      - Workflow Manager
      - Job
      - Notebook, Repos, DBSQL
    - - Workspace cluster
      - SQL DBW
- - - - Delta Lake table operations
        
        Time travel (Versionning)
        
        Versions
        
        SELECT .... FROM table_name VERSION AS OF 2
        
        RESTORE table_name FROM VERSION...
        
        SELECT ... FROM table_name v2
        
        Set up Delta Lake
        
        2, Create new tables from existing view/tables
        
        3, Declare table schema with generated columns + descriptive cmt
        
        4, Set options: location, constrains, partitions
        
        5, Creating clones: deep & shallow
        
        1, Create Delta tables using CTAS
        
        Load data into Delta Lake
        
        2, Overwrite data: INSERT OVERWRITE
        
        3, Append data: INSERT INTO
        
        4, Upsert (append, update, delete): MERGE INTO
        
        5, Ingest data incrementally: COPY INTO
        
        1, Overwrite tables: CRAS (CREATE OR REPLACE TABLE...)
        
        describe
        
        DESCRIBE DETAIL
        
        DESCRIBE HISTORY
        
        DESCRIBE EXTENDED
        
        Optimization
        
        OPTIMIZED table_name ZORDER BY col_name
        
        compact small files -> large files
        
        VACUUM
        
        -> no longer access to old version
        
        cleaning up stale files
        
        default 30 days auto clean
        
        Prevent delete less than 7 days -> Vacuum less than 7 days -> set ....retentionDurationCheck.enable = False
        
        VACUUM table_name RETAIN 12 HOURS
      - 2 types
        
        external Delta table
        
        managed Delta tables
    - - View
        
        persisted
      - Temporary View
        
        session scope
      - Global Temporary View
        
        cluster scope
        
        global_temp
- - - - Use case
        
        Workflows for every job
      - Components
        
        Tasks
        
        Define the task type
        
        Choose the cluster type
        
        Define task dependency if any
        
        Schedule
        
        Cluster
      - scheduling
        
        Type
        
        Manual
        
        Scheduled
        
        how to set up
        
        from UI
        
        cron syyntax
      - Alert
        
        email
        
        when
        
        Start
        
        Success
        
        Failure
      - Access Control
      - Job Run History
      - Repair a Failed Job Run
    - - Automated data pipelines for Delta Lake
      - Note: DLT pipeline can be a task in a workflow