Please enable JavaScript.
Coggle requires JavaScript to display documents.
S3 (Security, 5 methods of encrypting Objects, S3 Storage Classes, S3…
S3
Security
-
-
-
Networking
Supports VPC Endpoints without Internet
- Can use S3 bucket policies to control access to buckets from specific VPC endpoints
Logging and Audit
Features:
- S3 access logs can be stored in other S3 bucket (any request from any account, authorized or denied)
- Do not set logging bucket to be the monitored bucket to prevent a logging loop (Application bucket to be separated from Logging bucket)
- API calls can be logged in AWS CloudTrail
- S3 access logs can be analyzed using Amazon Athena
MFA Delete
Features:
- Must enable Versioning on S3 bucket to use MFA-Delete
- Use MFA to permanently delete object version
- Use MFA to suspend versioning on the bucket
- Only bucket owner (Root account) can enable/ disable MFA-Delete
- MFA-Delete currently can only be enabled using CLI
Pre-signed URLs
Features:
- URLs that are valid only for a limited time
- Use CLI for generating pre-signed URLs for downloads
- Use SDK for generating pre-signed URLs for uploads
- URL valid for 3600 seconds by default
- Can change timeout with --expires-in [TIME_BY_SECONDS] argument
- Users given a pre-signed URL inherit the permissions of the person who generated the URL for GET/PUT
Use cases:
- Allow only logged-in users to download a premium video on my S3 bucket
- Allow an ever changing list of users to download files by generating URLs dynamically
- Allow temporarily a user to upload a file to a precise location in our bucket
IAM principal can access S3 object if:
- User's IAM permissions allow it OR the resource policy allows it
- AND there is no explicit DENY
-
S3 Storage Classes
-
-
-
S3 Intelligent Tiering
Features:
- Low latency and high throughput performance of S3 Standard
- Small monthly monitoring and auto-tiering fee
- Automatically moves objects between 2 access tiers (S3 Standard - GP vs S3 Standard - IA) based on changing access patterns
- High durability (99.999999999%) of objects across multiple AZs
- 99.9% Availability over a given year
- Resilient against events that impact an entire AZ
- Minimum storage duration charge: 30 days
Amazon Glacier
Features:
- Low cost object storage meant for archiving/ backup
- Data is retained for longer term (10+ of years)
- Average annual durability is 99.999999999%
- Cost per storage per month ($0.004/GB) + retrieval cost
- Archive: item in Glacier (up to 40TB)
- Vaults: where Archive store
3 retrieval options:
- Expedited: 1 to 5 minutes
- Standard: 3 to 5 hours
- Bulk: 5 to 12 hours
- Minimum storage duration of 90 days
-
S3 Replication
Features:
- Must enable Versioning in source and destination
- Buckets can be in different accounts
- Asynchronous replication
- Must give proper IAM permissions to S3
-
-
Limitations:
- Only new Objects are replicated after activating (not retroactive)
- If delete without a Version ID, it adds a delete marker, not replicated
- If delete with a Version ID, it deletes in the source, not replicated
- No chaining of replication. If bucket 1 has replication into bucket 2, which has replication into bucket 3. Then Objects created in bucket 1 are not replicated to bucket 3
Lifecycle Rules
Transition actions
-
Examples:
- Move objects to Standard IA class 60 days after creation
- Move to Glacier for archiving after 6 months
Expiration actions
-
Examples:
- Access log files can be set to delete after a 365 days
- Can be used to delete old versions of files (if versioning is enabled)
- Can be used to delete incomplete multi-part uploads
Features:
- Rules can be created for certain prefix (example: s3://mybucket/mp3/*)
- Rules can be created for certain object tags (example: Department: Finance)
S3 Performance
Baseline Performance
Features:
- Automatically scales to high request rates, latency 100 - 200ms
- Application can achieve at least 3,500 PUT/COPY/POST/DELETE and 5500 GET/HEAD requests per second per prefix in a bucket
- No limits to the number of prefixes in a bucket
Prefix examples:
- bucket/myfolder1/mysubfolder1/myfile1
- bucket/myfolder2/mysubfolder2/myfile2
KMS Limitation (image)
Limitation:
- S3 calls GenerateDataKey KMS API when I upload file
- S3 calls Decrypt KMS API when I download file
- Count towards the KMS quota per second (5500, 10000, 30000 requests based on Region)
- Cannot request a quota increase for KMS
-
S3 Event Notifications
Examples:
- S3:ObjectCreated
- S3:ObjectRemoved
- S3:ObjectRestore
- Object name filtering
Features:
- Deliver events in seconds or minutes or longer
- If 2 writes are made to a single non-versioned object a the same time, it is possible that only 1 event notification will be sent
- Enable Versioning on bucket to ensure an event notification is sent for every successful write
- S3 supports the Lambda, SNS, SQS destinations where it can publish events
-
S3 Object Lock
Features:
- Must enable Versioning
- To block an object version deletion for a specified amount of time
Object Retention options:
- Retention Period: specifies a fixed period
- Legal Hold: same protection, no expiry date
Modes:
- Governance mode: user cannot overwrite or delete an object version or alter it lock settings unless they have special permissions
- Compliance mode: a protected object version cannot be overwritten or deleted by any user, including the root user in my AWS account. When an object is locked in compliance mode, its retention mode cannot be changed, and its retention period cannot be shortened
Buckets
Features:
- Buckets = directories
- S3 stores objects (files) in buckets
- Buckets is at Region level
Naming convention:
- No uppercase
- No underscore
- 3 - 63 characters long
- Not an IP
- Must start with lowercase letter or number
- Unique across AWS globally
Objects
Features:
- Objects have a Key
- Key is the FULL path
- Key is composed of <prefix> + <object name>
- Object values are the content of the body (maximum object size is 5TB)
- Must use Multi-part upload if uploading > 5GB at a time
- Support Metadata (Key/Value pair for System or User metadata)
- Support Tags (Unicode Key/Value pair - up to 10) for security/lifecycle
- Support Version ID if versioning is enabled
Examples:
- s3://my-bucket/my_file.txt
- s3://my-bucket/my-folder/another_folder/my_file.txt
- Prefix: my-folder/another_folder/
- Object name: my_file.txt
Versioning
Features:
- Enable at the Bucket level
- Same Key overwrite will increment the version
- Any file that is not versioned prior to enabling versioning will have version null
- Suspending versioning does not delete the previous versions
Pros:
- Protect against unintended deletes (ability to restore a version)
- Easy roll back to previous version
-
Athena
Features:
- Serverless service to perform analytics directly against S3 files
- Uses SQL to query
- Support JDBC/ ODBC driver
- Charge per query and amount of data scanned
- Supports CSV, JSON, ORC, Avro and Parquet (built on Presto)
Use cases: BI, Analytics, Reporting, Analyze & Query VPC Flow logs, ELB logs, CloudTrail trails
-
Static websites
URL will be:
- <bucket-name>.s3-website-<AWS-region>.amazonaws.com
- <bucket-name>.s3-website.<AWS-region>.amazonaws.com
Strong consistency
Features:
- new PUT (successful write)
- overwrite PUT (successful overwrite)
- DELETE (successful delete)
- read after write consistency
- list consistency
S3 Analytics
Features:
- Storage class analysis to determine when to transition objects from Standard to Standard_IA only
- Report is updated daily
- Take 24 to 48 hours to first start
-
S3 Requester Pays
Features:
- In general, bucket owners pay for all Amazon S3 storage and data transfer costs associated with their bucket
- With Requester Pays buckets, the requester instead of the bucket owner pays the cost of the request and data download from the bucket
- Requester must be authenticated
Glacier Vault Lock
Features:
- Adopt a Write Once Read Many model (WORM)
- Lock the policy for future edits
- Helpful for compliance and data retention