Please enable JavaScript.
Coggle requires JavaScript to display documents.
AWS Monitoring - Coggle Diagram
AWS Monitoring
CloudWatch
CloudWatch Metrics
Features:
- CloudWatch provides metrics for every service in AWS
- Metric is a variable to monitor (e.g. CPU utilization, Network utilization, Disk performance, and Disk Reads/Writes)
- Metrics belongs to Namespaces
- Dimension is an attribute of a metric (e.g. instance id, environment)
- Up to 10 dimensions per metric
- Metrics have timestamps
- Can create CloudWatch dashboards of metrics
Custom Metrics
Features:
- Can define and send my own custom metrics to CloudWatch
- Can use dimensions to segment metrics
- Instance.id
- Environment.name
- Metric resolution:
- Standard: 1 minute
- High Resolution: up to 1 second (StorageResolution API parameter) - Higher cost
- Use API call PutMetricData to send metric
- Use exponential back off in case of throttle errors
EC2 Detailed monitoring:
- EC2 instance metrics have updated every 5 minutes
- Can get data every 1 minutes with detailed monitoring (for a cost)
- Use detailed monitoring if I want to more prompt scale my ASG
- Free Tier allows us to have 10 detailed monitoring metrics
- EC2 Memory usage is by default not pushed (must be pushed from inside the instance as a custom metric)
CloudWatch Dashboard:
- Dashboards are Global
- Can include graphs from different Regions
- Can change time zone & time range of the dashboards
- Can setup automatic refresh (10s, 1m, 2m, 5m, 15m)
- Pricing:
- 3 dashboards (up to 50 metrics) for free
- $3 per dashboard per month afterwards
CloudWatch Logs
Features:
- Application can send logs to CloudWatch using the SDK
- CloudWatch can collect log from:
- Elastic Beanstalk: collection of logs from application
- ECS: collect from containers
- Lambda: collect from function logs
- VPC Flow Logs: VPC specific logs
- API Gateway
- CloudTrail based on filter
- CloudWatch Logs Agent: e.g. EC2 machine
- Route 53: Log DNS queries
- CloudWatch Logs can go to:
- Batch exporter to S3 for archival
- Stream to ElasticSearch cluster for further analytics
- Logs storage architecture:
- Log groups: arbitrary name, usually representing an application
- Log stream: instances within application/ log files/ containers
- Can define log expiration policies (never expire, 30 days)
- Use CLI we can tail CloudWatch logs
- Need the right IAM permission to send logs to CloudWatch
- Security: encryption of logs using KMS at the group level
- Can use filter expressions:
- e.g. find a specific IP inside of a log
- Can be used to trigger alarms
- CloudWatch Logs Insights can be used to query logs and add queries to Dashboards
CloudWatch Logs for EC2
Features:
- By default no logs from EC2 will go to CloudWatch
- Need CloudWatch Logs Agent to push the log files to CloudWatch
- Need the right IAM permissions
- CloudWatch Logs Agent can be setup on-premise (image)
-
CloudWatch Alarm
Features:
- Use to trigger notifications for any metric
- Alarms can go to ASG, EC2 actions, SNS notifications
- Various options (sampling, %, max, min)
- Alarm States: OK, INSUFFICIENT_DATA, ALARM
- Period:
- Length of time in seconds to evaluate the metric
- High resolution custom metrics: can only choose 10 sec or 30 sec
EC2 Instance Recovery:
- Use StatusCheckFailed_System (image)
- After recovery, same Private, Public, Elastic IP, metadata, placement group
CloudWatch Events:
- Use Cron jobs to schedule events in CloudWatch
- Use to create Event Pattern: Event rules to reach to a service doing something (e.g. CodePipeline state changes)
- Triggers to Lambda functions, SQS/SNS/Kinesis messages
- CloudWatch Event creates a small JSON document to give information about the change
CloudTrail Event
-
CloudTrail Events Retention:
- Events are stored for 90 days in CloudTrail
- To keep events beyond this period, log them to S3 and use Athena
Features:
- Provides governance, compliance, inspect audit for my AWS account
- Enable by default
- Get an history of events/ API calls made within my AWS account by Console, SDK, CLI, AWS Services
- Can put logs from CloudTrail into CloudWatch Logs or S3
- By default, CloudTrail event log files are encrypted using S3 SSE
- A trail can be applied to All Regions (by default) or single Region
- CloudTrail diagram (image)
AWS Config
Features:
- Helps with auditing and recording compliance of my AWS resources
- Helps record configurations and changes over time
- Can store the configuration data into S3 (analyzed by Athena)
- Examples:
- Is there unrestricted SSH access to my security groups?
- Do my buckets have public access?
- How has my ALB configuration changed over time?
- Can receive alerts (SNS notifications) for any changes
- AWS Config is a per-region service
- Can be aggregated across regions and accounts
Config Resources:
- View compliance of a resource over time
- View configuration of a resource over time
- View CloudTrail API calls if enabled
Config Rules:
- Can use AWS managed config rules (over 75)
2, Can make custom config rules (must be defined in Lambda). For examples:
- Evaluate if each EBS disk is of type gp2
- Evaluate if each EC2 instance is t2.micro
- Rules can be evaluated/ triggered:
- For each config change
- And/ Or at regular time intervals
- Can trigger CloudWatch Events if the rule is non-compliant (and chain with Lambda)
- Rules can have auto remediation:
- If a resource is not compliant, you can trigger an auto remediation (e.g. stop instances with non-approved tags)
- Does not prevent actions from happening (have to remediate to that)
- Pricing: no Free Tier, $2 per active rule per Region per month
CloudWatch vs CloudTrail vs AWS Config (image)
- ELB example