Please enable JavaScript.

Coggle requires JavaScript to display documents.

Operations (Software attributes (Reliability (Consider early, hard to…

- - - - "Monitoring is for symptom based Alerting" (SRE book)
      - answers
        
        why is it broken?
        
        what is broken?
      - consists of
        
        black-box monitoring
        
        good for "what is broken?"
        
        white-box monitoring
        
        good to anticipate what will eventually break if nothing done
        
        good to cover "known failure modes"
      - how?
        
        online-serving systems (microservice, DB, etc.)
        
        number of queries
        
        number of errors
        
        latency
        
        both client and server side
        
        offline systems (batch, M/R, etc.)
        
        for each stage
        
        number of items coming in
        
        number of items in progress
        
        number of items going out
        
        latest heartbeat
        
        overall runtime
        
        last completed job time
        
        misc.
        
        threadpools
        
        number of queued items
        
        number of threads running
        
        total number of threads
        
        number of items processed
        
        how long to process an item
        
        how long to wait in the queue
        
        number of log messages (for each level)
        
        caches
        
        total number of queries
        
        number of hits
        
        latency
        
        "no part of a metric name should ever be procedurally generated (use labels instead)" but "try to keep the cardinality of your metrics below 10"
    - - message ID: each message has an UUID
      - causation ID: the UUID of the incoming message
      - correlation ID: an UUID which is copied from incoming message to subsequent, related messages, responses, etc.