Please enable JavaScript.

Coggle requires JavaScript to display documents.

DP-203 - Chapter 13 - Monitoring Data Storage and Data Processing, …

- - - - time-series db
        numerical values
        recorded at regular intervals
      - can be aggreated
        
        alerting
        
        reporting
        
        auditing
    - - event driven
        textual
  - - - Log Analytics Workspace
        
        Kusto Query Language (KQL)
- - - - hardware, memory, disk, CPU, ...
    - - storage accounts, Azure SQL, ...
        
        data readily captured
      - Metrics page
        
        service specific metrics
        
        ingress,
        
        egress,
        
        blob capacity
        
        ...
      - Azure Monitor
        
        services:
        
        VM
        
        Storage accounts ->
        
        SQL
        
        Key Vaults
        
        overall service performance
        
        Availability
        
        Latency
        
        ...
        
        Insights
        
        Failures
        
        Performance
        
        Availability
        
        Capacity
    - - Account usage, ...
    - - AAD, ...
    - - custom built-applications on top of Azure
        
        install Azure Monitor Agent (AMA) on host
- - - - diagnostic details of activities within pipeline
      - overall consumption at pipeline level
- - - - Parquet : on by default
    - - CSV
      - External tables
    - - DROP STATISTICS
        
        CREATE STATISTICS
  - - - ALTER DATABASE [DW_NAME] SET AUTO_CREATE_STATISTICS ON
      - :warning: not available for temporary or external tables
    - - CREATE STATISTICS [statistics_name]
        ON [schema_name].[table_name]([column_name])
        WITH SAMPLE 40 PERCENT;
      - :check:
        
        < 1 billion rows => default sampling (20%).
        
        > 1 billion rows => sampling of 2 to 10%.
- - - - Setup and run TPC-H and TPC-DS
        benchmarking suites on regular basis
    - - Metrics tab
        
        Data Warehouse Units
        Memory usage
      - System tables
        
        sys.dm_pdw_exec_requests
        sys.dm_pdw_waits
        
        query wait times
        elapsed time
      - Query store
        
        ALTER DATABASE <database_name> SET QUERY_STORE = ON;
        
        Query details
        
        Plan details
        
        Runtime statistics
- - - - Dashboard
        
        Monitor overall cluster performance
        
        heap usage
        
        disk usage
        
        CPU usage
        
        network latencies
      - Hosts
        
        Monitor per-node performance
        
        disk usage
        
        CPU usage
        
        memory usage
        
        network usage
        
        ...
        
        Drill down to query level
      - Resource Manager
        
        Application Queues
        
        Scheduler
        
        Monitor at query level
        
        status of YARN queues
  - - - check storage accounts
        
        increase limits
        
        distribute processing over multiple storage accounts