Please enable JavaScript.
Coggle requires JavaScript to display documents.
Data Storage - Coggle Diagram
Data Storage
S3
S3 Buckets:
- S3 allows people to store objects (files) in buckets directories)
- Buckets must have a globally unique name
- Buckets are defined at the region level
- Naming convention:
- No uppercase
- No underscore
- 3 to 63 characters long
- Not an IP
- Must start with lowercase letter or number
S3 Objects:
- Objects (files) have a Key. The key is the FULL path:
- <my_bucket>/my_file.txt
- <my_bucket>/my_folder1/ another_folder my_file.txt
- There’s no concept of “directories” within buckets (although the UI will trick you to think
- Just keys with very long names that contain slashes (“/”)
- Object Values are the content of the body:
- Max Size is 5TB
- Must use “multi part upload” if uploading more than 5GB
- Metadata (list of text key/ value pairs system or user metadata)
- Tags (Unicode key/ value pair up to 10) useful for security/ lifecycle
- Version ID (if versioning is enabled)
S3 Consistency Model:
- After a:
- successful write of a new object (new PUT)
- or an overwrite or delete of an existing object (overwrite PUT or DELETE)
- any:
- subsequent read request immediately receives the latest version of the object (read after write consistency)
- subsequent list request immediately reflects changes (list consistency)
- Available at no additional cost, without any performance impact
S3 Storage Classes:
- S3 Standard General Purpose
- S3 Standard Infrequent Access (IA)
- S3 One Zone Infrequent Access
- S3 Intelligent Tiering
- Amazon Glacier
- Amazon Glacier Deep Archive
- S3 Storage Classes Comparison (image)
- S3 Storage Classes Price Comparison (image)
S3 Standard General Purpose:
- High durability (99.999999999%) of objects across multiple AZ
- If I store 10,000,000 objects with S3, I can on average expect to incur a loss of a single object once every 10,000 years
- 99.99% Availability over a given year
- Sustain 2 concurrent facility failures
- Use Cases: Big Data analytics, mobile & gaming applications, content distribution
S3 Standard Infrequent Access (IA):
- Suitable for data that is less frequently accessed, but requires rapid access when needed
- High durability (99.999999999%) of objects across multiple AZs
- 99.9% Availability
- Low cost compared to S3 Standard
- Sustain 2 concurrent facility failures
- Use Cases: As a data store for disaster recovery, backups
S3 One Zone Infrequent Access (IA):
- Same as IA but data is stored in a single AZ
- High durability (99.999999999%) of objects in a single AZ; data lost when AZ is destroyed
- 99.5% Availability
- Low latency and high throughput performance
- Supports SSL for data at transit and encryption at rest
- Low cost compared to IA (by 20%)
- Use Cases: Storing secondary backup copies of on premise data, or storing data I can recreate
S3 Intelligent Tiering:
- Same low latency and high throughput performance of S3 Standard
- Small monthly monitoring and auto tiering fee
- Automatically moves objects between two access tiers based on changing access patterns
- Designed for durability of 99.999999999% of objects across multiple Availability Zones
- Resilient against events that impact an entire Availability Zone
- Designed for 99.9% availability over a given year
Amazon Glacier:
- Low cost object storage meant for archiving/ backup
- Data is retained for the longer term (10s of years)
- Alternative to on premise magnetic tape storage
- Average annual durability is 99.999999999%
- Cost per storage per month ($0.004 / GB) + retrieval cost
- Each item in Glacier is called Archive (up to 40 TB)
- Archives are stored in Vaults
Amazon Glacier:
- 3 retrieval options:
- Expedited (1 to 5 minutes) - $0.03 per GB and $0.01 per request
- Standard (3 to 5 hours) - $0.01 per GB and $0.05 per 1,000 requests
- Bulk (5 to 12 hours) - $0.0025 per GB and $0.0025 per 1,000 requests
- Minimum storage duration of 90 days
Glacier Deep Archive:
- for long term storage - cheaper
- Standard (12 hours)
- Bulk (48 hours)
- Minimum storage duration of 180 days
Glacier Vault Policies & Vault Lock:
- Vault is a collection of archives
- Each Vault has:
- 1 vault access policy
- 1 vault lock policy
- Vault Policies are written in JSON
- Vault Access Policy is similar to Bucket Policy (restrict user/ account permissions)
- Vault Lock Policy is a policy I lock for regulatory and compliance requirements
- Policy is immutable, it can never be changed
- e.g. forbid deleting an archive if < 1 year old
- e.g. implement WORM policy (write once read many)
S3 Moving between storage classes:
- For infrequently accessed object, move them to STANDARD_IA
- For archive objects I don’t need in real time, GLACIER or DEEP_ARCHIVE
- Moving objects can be automated using a lifecycle configuration
S3 Lifecycle Rules:
- Transition actions:
- Defines when objects are transitioned to another storage class
- e.g. Move objects to Standard IA class 60 days after creation
- e.g. Move to Glacier for archiving after 6 months
- Expiration actions:
- Configure objects to expire (delete) after some time
- Access log files can be set to delete after a 365 days
- Can be used to delete old versions of files (if versioning is enabled)
- Can be used to delete incomplete multi part uploads
- Rules can be created for a certain prefix (e.g. s3://mybucket/mp3/*)
- Rules can be created for certain objects tags (e.g. Department: Finance)
S3 Versioning:
- Can version my files in S3
- It is enabled at the bucket level
- Same key overwrite will increment the Version
- It is best practice to version your buckets
- Protect against unintended deletes (ability to restore a version)
- Easy roll back to previous version
- Any file that is not versioned prior to enabling versioning will have version Null
- Suspending versioning does not delete the previous versions
S3 Replication (CRR & SRR):
- Must enable versioning in source and destination
- Cross Region Replication (CRR)
- Same Region Replication (SRR)
- Buckets can be in different accounts
- Copying is asynchronous
- Must give proper IAM permissions to S3
- CRR - Use cases: compliance, lower latency access, replication across accounts
- SRR - Use cases: log aggregation, live replication between production and test accounts
- After activating, only new objects are replicated (not retroactive)
- For DELETE operations:
- Can replicate delete markers from source to target (optional setting)
- There is no chaining of replication
- If bucket 1 has replication into bucket 2, which has replication into bucket 3
- Then objects created in bucket 1 are not replicated to bucket 3
How to verify the file is exactly the same as the 1 that I have uploaded?
- Use S3 ETag (Entity Tag)
- For simple uploads (< 5GB), it has the MD5 hash
- we can ensure integrity of files using ETag
S3 Performance
S3 KMS Limitation:
- I may be impacted by the KMS limits, if you use SSE-KMS
- When I upload, it calls the GenerateDataKey KMS API
- When I download, it calls the Decrypt KMS API
- Count towards the KMS quota per second (5500, 10000, 30000 request per second based on region) (image)
- Limit by how many times can call the SSE-KMS service
- Can request a quota increase using the Service Quotas Console
S3 Baseline Performance:
- S3 automatically scales to high request rates, latency 100 - 200ms
- My application can achieve at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per prefix in a bucket
- There are no limits to the number of prefixes in a bucket
- For example (object path => prefix):
- e.g. bucket/folder1/sub1/file => /folder1/sub1/
- e.g. bucket/folder1/sub2/file => /folder1/sub2/
- e.g. bucket/1/file => /1/
- e.g. bucket/2/file => /2/
- I can achieve 22,000 requests per second for GET and HEAD if I spread reads across all four prefixes evenly
Improve S3 Performance:
- Multi-Part upload: (image)
- Recommended for files > 100MB, must use for files > 5GB
- Can help parallelize uploads (speed up transfers)
- S3 Transfer Acceleration: (image)
- Increase transfer speed by transferring file to an AWS edge location which will forward the data to the S3 bucket in the target region
- Compatible with multi-part upload
- S3 Byte Range Fetches:
- Parallelize GETs by requesting specific byte ranges
- Better resilience in case of failures
- Can be used to speed up downloads (image)
- Can be used to retrieve only partial data (e.g. the head of a file)
4 methods of encrypting objects in S3:
- SSE-S3: encrypts S3 objects using keys handled & managed by AWS
- SSE-KMS: leverage AWS Key Management Service to manage encryption keys
- SSE-C: when I want to manage my own encryption keys
- Client Side Encryption
SSE-S3:
- encryption using keys handled & managed by S3
- Object is encrypted server side
- AES-256 encryption type
- Must set header: x-amz-server-side-encryption: AES256
- S3 manage the data and the encryption keys (image)
SSE-KMS:
- SSE-KMS: encryption using keys handled & managed by KMS
- KMS Advantages: user control + audit trail
- Object is encrypted server side
- Must set header: x-amz-server-side-encryption: aws:kms
- AWS manage the data key but I manage the customer master key (CMK) in AWS KMS (image)
SSE-C:
- SSE-C: server-side encryption using data keys fully managed by the customer outside of AWS
- S3 does not store the encryption key I provide
- HTTPS must be used
- Encryption key must provided in HTTP headers, for every HTTP request made
- I manage the encryption key and S3 manages the encryption (image)
Client Side Encryption:
- Client library such as the Amazon S3 Encryption Client
- Clients must encrypt data themselves before sending to S3
- Clients must decrypt data themselves when retrieving from S3
- Customer fully manages the keys and encryption cycle
Encryption in transit (SSL/TLS):
- Amazon S3 exposes:
- HTTP endpoint: non encrypted
- HTTPS endpoint: encryption in flight
- HTTPS is recommended
- HTTPS is mandatory for SSE-C
- Encryption in flight is also called SSL / TLS
S3 Cross-Origin Resource Sharing (CORS): (image)
- Need to enable CORS if I request data from another website
- CORS allows me to limit the number of websites that can request my files in S3 (and limit my costs)
S3 Security:
- User based:
- IAM policies which API calls should be allowed for a specific user from IAM console
- Resource Based:
- Bucket Policies bucket wide rules from the S3 console allows cross account
- Object Access Control List (ACL) - finer grain
- Bucket Access Control List (ACL) - less common
- IAM principal can access an S3 object if
- the user IAM permissions allow it OR the resource policy allows it AND there’s no explicit DENY
- Networking:
- Supports VPC Endpoints (for instances in VPC without www)
- Logging and Audit:
- S3 Access Logs can be stored in other S3 bucket
- API calls can be logged in AWS CloudTrail
- User Security:
- MFA Delete: MFA (multi factor authentication) can be required in versioned buckets to delete objects
- Pre-Signed URLs: URLs that are valid only for a limited time (e.g. premium video service for logged in users)
S3 Bucket Policies:
- JSON based policies:
- Resources: buckets and objects
- Actions: Set of API to Allow or Deny
- Effect: Allow / Deny
- Principal: The account or user to apply the policy to
- Use S3 bucket for policy to:
- Grant public access to the bucket
- Force objects to be encrypted at upload
- Grant access to another account (Cross Account)
Bucket settings for Block Public Access:
- Block public access to buckets and objects granted through
- new access control lists (ACLs)
- any access control lists (ACLs)
- new public bucket or access point policies
- Block public and cross account access to buckets and objects through any public bucket or access point policies
- These settings were created to prevent company data leaks
- Can be set at the account level
S3 Select & Glacier Select: (image)
- Retrieve less data using SQL by performing server-side filtering
- Can filter by rows & columns (simple SQL statements)
- Less network transfer, less CPU cost client-side
- Glacier Select can only do uncompressed CSV files
- Use cases: S3 Select with Hadoop
- Filter data from S3 before analyzing it with my cluster (image)
S3 Event Notifications:
- S3:ObjectCreated, S3:ObjectRemoved, S3:ObjectRestore, S3:Replication
- Object name filtering possible (*.jpg)
- Use case: generate thumbnails of images uploaded to S3
- Can create as many "S3 events" as desired
- S3 event notifications typically deliver events in seconds but can sometimes take a minute or longer
- If two writes are made to a single non versioned object at the same time, it is possible that only a single event notification will be sent
- If I want to ensure that an event notification is sent for every successful write, I can enable versioning on my bucket
DynamoDB
Features:
- Fully Managed, Highly available with replication across 3 AZs
- NoSQL database not a relational database
- Scales to massive workloads, distributed database
- Millions of requests per seconds, trillions of row, 100s of TB of storage
- Fast and consistent in performance (low latency on retrieval)
- Integrated with IAM for security, authorization and administration
- Enables event driven programming with DynamoDB Streams
- Low cost and auto scaling capabilities
- DynamoDB in Big Data (image)
- DynamoDB is going to be more for when my data is hot and smaller while S3 is going to be more when data is cold and bigger
- Other features:
- Support Point in time restore like RDS (No performance impact)
- Support multi-region, fully replicated, high performance Global Tables
- Can launch a local DynamoDB on my computer for development purposes
- Use Amazon Database Migration Service (DMS) to migrate to DynamoDB (from Mongo, Oracle, MySQL, S3)
DynamoDB Basics:
- Made of tables
- Each table has a primary key (must be decided at creation time)
- Each table can have an infinite number of items (= rows)
- Each item has attributes (can be added over time can be null)
- Maximum size of a item is 400KB
- Data types supported are:
- Scalar Types: String, Number, Binary, Boolean, Null
- Document Types: List, Map
- Set Types: String Set, Number Set, Binary Set
DynamoDB Primary Keys:
- Option 1: Partition key only (HASH):
- Partition key must be unique for each item
- Partition key must be “diverse” so that the data is distributed
- e.g. user_id for a users table (image)
- Option 2: Partition Key + Sort Key:
- The combination must be unique
- Data is grouped by partition key
- Sort key == range key
- e.g. users games table (image)
- user_id for the partition key
- game_id for the sort key
DynamoDB Provisioned Throughput:
- Table must have provisioned read and write capacity units
- Read Capacity Units (RCU): throughput for reads
- Write Capacity Units (WCU): throughput for writes
- Option to setup auto scaling of throughput to meet demand
- Throughput can be exceeded temporarily using “burst credit”
- If burst credit are empty, I will get a ProvisionedThroughputException
- It’s then advised to do an exponential back off retry
DynamoDB Write Capacity Units:
- 1 WCU represents 1 write per second for an item up to 1 KB in size
- If the items are > 1 KB, more WCU are consumed
- e.g. we write 10 objects per seconds of 2 KB each
- e.g. we write 6 objects per second of 4.5 KB each
- We need 6 x 5 = 30 WCU (4.5 gets rounded to the upper KB)
- e.g. we write 120 objects per minute of 2 KB each
- We need 120 / 60 x 2 = 4 WCU
Strongly Consistent Read vs Eventually Consistent Read:
- Eventually Consistent Read: If we read just after a write, it is possible we will get unexpected response because of replication (image)
- Strongly Consistent Read: If we read just after a write, we will get the correct data
- By default: DynamoDB uses Eventually Consistent Reads, but GetItem, Query & Scan provide a ConsistentRead ” parameter you can set to True
DynamoDB Read Capacity Units:
- 1 RCU represents 1 strongly consistent read per second, or two eventually consistent reads per second, for an item up to 4 KB in size
- If the items are > 4 KB, more RCU are consumed
- e.g. 10 strongly consistent reads per seconds of 4 KB each
- We need 10 / 4 KB x 4 KB = 10 RCU
- e.g. 16 eventually consistent reads per seconds of 12 KB
- We need (16 / 2) * (12 / 4 KB) = 24 RCU
- e.g. 10 strongly consistent reads per seconds of 6 KB each
- We need 10 x 8 KB / 4 KB = 20 RCU
DynamoDB Throttling:
- If we exceed our RCU or WCU, we get ProvisionedThroughputExceededExceptions
- Reasons:
- Hot keys / partitions: one partition key is being read too many times
- Very large items: remember RCU and WCU depends on size of items
- Solutions:
- Exponential back off when exception is encountered (already in SDK)
- Distribute partition keys as much as possible
- If RCU issue, we can use DynamoDB Accelerator (DAX)
How to calculate the number of partitions?
- Start with one partition
- Each partition:
- Max of 3000 RCU / 1000 WCU
- Max of 10GB
- To compute the number of partitions:
- By capacity: (TOTAL RCU / 3000) + (TOTAL WCU / 1000)
- By size: Total Size / 10 GB
- Total partitions = CEILING(MAX(Capacity, Size))
- WCU and RCU are spread evenly between partitions (image)
DynamoDB API
Writing Data:
- PutItem: Write data to DynamoDB (create data or full replace)
- UpdateItem: Update data in DynamoDB (partial update of attributes)
- Possibility to use Atomic Counters and increase them
- Conditional Writes:
- Accept a write / update only if conditions are respected, otherwise reject
- Helps with concurrent access to items
- No performance impact
Deleting Data:
- DeleteItem:
- Delete an individual row
- Ability to perform a conditional delete
- DeleteTable:
- Delete a whole table and all its items
- Much quicker deletion than calling DeleteItem on all items
Batching Writes:
- BatchWriteItem:
- Up to 25 PutItem and / or DeleteItem in 1 call
- Up to 16 MB of data written
- Up to 400 KB of data per item
- Batching allows me to save in latency by reducing the number of API calls done against DynamoDB
- Operations are done in parallel for better efficiency
- Possible for part of a batch to fail, in which case we have the try the failed items (using exponential back off algorithm)
Reading Data:
- GetItem:
- Read based on Primary key
- Primary Key = HASH or HASH RANGE (partition key + sort key)
- Eventually consistent read by default
- Option to use strongly consistent reads (more RCU might take longer)
- ProjectionExpression can be specified to include only certain attributes (filter certain data)
- BatchGetItem:
- Up to 100 items OR
- Up to 16 MB of data
- Items are retrieved in parallel to minimize latency
Query:
- Query returns items based on:
- PartitionKey value ( must be = operator)
- SortKey value (=, <, <=, >, >=, Between, Begin) optional
- FilterExpression to further filter (client side filtering)
- Returns:
- Up to 1 MB of data
- Or number of items specified in Limit
- Able to do pagination on the results
- Can query table, a local secondary index, or a global secondary index
Scan:
- Scan the entire table and then filter out data (inefficient)
- Returns up to 1 MB of data use pagination to keep on reading
- Consumes a lot of RCU
- Limit impact using Limit or reduce the size of the result and pause
- For faster performance, use parallel scans:
- Multiple instances scan multiple partitions at the same time
- Increases the throughput and RCU consumed
- Limit the impact of parallel scans just like you would for Scans
- Can use a ProjectionExpression + FilterExpression (no change to RCU)
DynamoDB Indexing
DynamoDB Local Secondary Index (LSI): (image)
- Alternate range key for my table, local to the hash key
- Up to five local secondary indexes per table
- Sort key consists of exactly one scalar attribute
- Attribute that you choose must be a scalar String, Number, or Binary
- LSI must be defined at table creation time
DynamoDB Global Secondary Index (GSI): (image)
- To speed up queries on non-key attributes, use a GSI
- GSI = partition key + optional sort key
- The index is a new “table” and we can project attributes on it
- The partition key and sort key of the original table are always projected (KEYS_ONLY)
- Can specify extra attributes to project (INCLUDE)
- Can use all attributes from main table (ALL)
- Must define RCU / WCU for the index
- Possibility to add / modify GSI (but not LSI)
DynamoDB DAX: (image)
- DAX = DynamoDB Accelerator
- Seamless cache for DynamoDB, no application re-write
- Writes go through DAX to DynamoDB
- Micro second latency for cached reads & queries
- Solves the Hot Key problem (too many reads)
- 5 minutes TTL for cache by default
- Up to 10 nodes in the cluster
- Multi AZs (3 nodes minimum recommended for production)
- Secure (Encryption at rest with KMS, VPC, IAM, CloudTrail)
DynamoDB Streams
Streams with Lambda: (image)
- Changes in DynamoDB (Create, Update, Delete) can end up in a DynamoDB Stream
- Stream can be read by Lambda, and we can then do:
- React to changes in real time (welcome email to new users)
- Create derivative tables / views
- Insert into ElasticSearch
- Could implement Cross Region Replication (CRR) using Streams
- Stream has 24 hours of data retention
- Configurable batch size received by Lambda (up to 1,000 rows, 6 MB)
Streams with Kinesis Adapter: (image)
- Alternative to using Lambda
- Use the KCL library to directly consume from DynamoDB Streams
- I just need to add a “Kinesis Adapter” library
- The interface and programming is exactly the same as Kinesis Streams
DynamoDB Time to Live (TTL):
- TTL = automatically delete an item after an expiry date / time
- TTL is provided at no extra cost, deletions do not use WCU / RCU
- TTL is a background task operated by the DynamoDB service itself
- Helps reduce storage and manage the table size over time
- Helps adhere to regulatory norms
- TTL is enabled per row (I define a TTL column, and add a date there)
- DynamoDB typically deletes expired items within 48 hours of expiration
- Deleted items due to TTL are also deleted in GSI / LSI
- DynamoDB Streams can help recover expired items
- https://www.epochconverter.com/
DynamoDB Security:
- Use VPC Endpoints to access DynamoDB without internet
- Access fully controlled by IAM
- Encryption at rest using KMS
- Encryption in transit using SSL / TLS
DynamoDB + S3 Storing large objects:
- Max size of item in DynamoDB = 400 KB
- For large objects, store them in S3 and reference them in DynamoDB (image)
- DynamoDB Only vs DynamoDB + S3 Price Comparison for 300 KB (image)
AWS ElastiCache:
- The same way RDS is to get managed Relational Databases
- ElastiCache is to get managed Redis or Memcached
- Caches are in memory databases with really high performance, low latency
- Helps reduce load off of databases for read intensive workloads
- Helps make your application stateless
- Write Scaling using Sharding
- Read Scaling using Read Replicas
- Multi AZ with Failover Capability
- AWS takes care of OS maintenance / patching, optimizations, setup, configuration, monitoring, failure recovery and backups
Redis Overview:
- Redis is an in memory key value store
- Super low latency (sub ms)
- Cache survive reboots by default (it’s called persistence)
- Use cases:
- User sessions
- Leaderboard (for gaming)
- Distributed states
- Relieve pressure on databases (such as RDS)
- Pub / Sub capability for messaging
- Multi AZ with Automatic Failover for disaster recovery if I don’t want to lose my cache data
- Support for Read Replicas
Memcached Overview:
- Memcached is an in memory object store
- Cache doesn’t survive reboots
- Use cases:
- Quick retrieval of objects from memory
- Cache often accessed objects
- Overall, Redis has largely grown in popularity and has better feature sets than Memcached