Please enable JavaScript.
Coggle requires JavaScript to display documents.
Storage - Amazon S3 - Coggle Diagram
Storage - Amazon S3
replication
= enables automatic, asynchronous copying of objects across S3 buckets
buckets that are configured for object replication can be owned by the same AWS account or by different accounts
-
by default, replication only supports copying new S3 objects after it is enabled, existing objects can be replicated through Support request
to enable, add replication configuration to source bucket
-
-
-
requirements
-
-
source bucket owner must have the source and destination regions enabled in account, destination bucket owner must have destination region enabled in account
if owner of the source bucket doesn't own the object then object owner must grant bucket owner READ and READ_ACP permissions with the object ACL
if source bucket has S3 Object Lock enabled, destination buckets must also have S3 Object Lock enabled
what is replicated
-
-
-
-
-
object ACL updates, unless you direct S3 to change the replica ownership when source and destination buckets aren't owned by the same accounts
-
-
delete and replication
if DELETE request without specifying an object version ID, Amazon S3 adds a delete marker
if latest version of the replication configuration (= with Filter element in rule) then no replication of marker by default but can be enabled
if not latest version of replication configuration, then replication of marker that resulted from user action, but not if due to lifecycle action
if DELETE request with object version ID, S3 deletes object version in source bucket
-
-
what is not replicated
-
objects in the source bucket that are replicas that were created by another replication rule (no replication chaining)
-
-
-
objects in the source bucket that the bucket owner doesn't have permissions for (when the bucket owner is not the owner of the object)
-
-
objects
-
-
-
-
-
-
multipart upload, in general, when object size reaches 100 MB, consider using multipart uploads instead of uploading the object in a single operation
-
three-step process
-
parts upload (up to 10000 parts, give upload ID + part number and receive ETag)
-
-
copy
-
-
object greater than 5 GB, use the multipart upload API
-
deletion
-
-
object version-enabled
if non-versioned delete request (= only specify key name) then S3 creates delete marker and returns marker's version ID, object disappears from bucket
if versioned delete request (= specify both key and version ID) then specific version deleted OR if version points to marker then marker deleted (making the object reappear in the bucket)
MFA-enabled buckets
-
-
-
-
BUT: multi-object delete request succeeds when specifying only non-versioned objects and no MFA token provided
buckets
name is globally unique, namespace shared by all AWS accounts in standard regions (but: partitions; meaning China and GovCloud are actually separate namespaces)
-
-
-
encryption
-
customer master keys (CMKs) stored in AWS KMS (SSE-KMS); can also enable S3 Bucket Keys to reduce cost of encryption
-
-
to encrypt existing S3 objects with a single request, you can use Amazon S3 Batch Operations
S3 inventory
-
provides CSV, ORC or Parquet output files that list objects and metadata on daily or weekly basis for a bucket or shared prefix
-
can configure what object metadata to include in the inventory, whether to list all object versions or only current versions, where to store the inventory list file output
can query S3 inventory using standard SQL by using Athena, Redshift Spectrum, and other tools such as Presto, Hive and Spark
-
-
S3 Select
-
works on objects stored in CSV, JSON, or Apache Parquet format
also works with objects that are compressed with GZIP or BZIP2 (for CSV and JSON objects only), and server-side encrypted objects
-
-
-
storage classes
-
-
-
for archiving objects
S3 Glacier = has a minimum storage duration period of 90 days and can be accessed in as little as 1-5 minutes using expedited retrieval
S3 Glacier Deep Archive = has a minimum storage duration period of 180 days and a default retrieval time of 12 hours
-
objects remain in S3 when using Glacier as storage class, cannot be accessed through separate S3 Glacier service
for S3 on Outposts
S3 Outposts
-
can also explicitly choose to encrypt objects using SSE-C (SSE with customer-provided encryption keys
presigned URLs
-
use a presigned URL to optionally share objects or enable customers/users to upload objects to buckets without AWS security credentials or permissions
-
in essence, presigned URLs are a bearer token that grants access to customers who possess them
S3 Batch Operations
-
-
-
can also initiate object restores from S3 Glacier or invoke Lambda function to perform custom actions using your objects
-
-
S3 Block Public Access
= provides settings for access points, buckets, and accounts to manage public access to S3 resources
by default no public access on buckets, objects and access points
-
S3 Access Points
-
= named network endpoints that are attached to buckets that you can use to perform S3 object operations
enforces a customized access point policy that works in conjunction with the bucket policy that is attached to the underlying bucket
analytics and insights
-
S3 Storage Lens
aggregates usage and activity metrics and displays the information in an interactive dashboard or through a metrics data export in CSV or Parquet format
visualize insights and trends, flag outliers
-
-
-
-
-
-