Please enable JavaScript.

Coggle requires JavaScript to display documents.

Data Migration, RedShift, AWS Elasticsearch Service, Glue, Elastic Map…

- - - - Create DMS endpoints & tasks
      - OLTP & OLAP schema conversions
- - - - Lucene index of its own
      - Might be on a different node in cluster
- - - - Default - stored in master node MySQL
      - External metastore - AWS Glue Data Catalog or AWS RDS
    - - Load partitions or scripts from S3
      - Write tables in S3
      - DynamoDB as an external table
- - - - KMS - works with root volume
      - LUKS - doesn't work with root volume
- - - - Alternate to range key, local to hash key
      - Upto 5 LSIs / table
      - Sort key - 1 scalar attribute
      - Must be defined at table creation
    - - Partition key + optional sort key
      - Possible to add / modify GSI
      - Speed up queries on non-key attributes
  - - - Strongly consistent 1 read/sec for 4KB
      - Eventually consistent 2 read/sec for 4KB
    - - 1 write/sec for 1KB object
    - - ProvisionedThroughputExceededException
        
        Use "burst credit" to increase throughput
        
        Exponential back-off retry
        
        If RCU issue -> use DyanmoDB accelerator (DAX)
    - - GetItem, Query & Scan -> "ConsistentRead" possible
- - - - Shard splitting - divide a hot shard
      - Merging shards - save costs, group shards with low traffic
      - Auto-scaling: implemented with AWS Lambda using UpdateShardCount
      - Resharding cannot be done in parallel
      - Low latency - 70 ms (enhanced fan-out) & 200 ms (classic)
- - - - Reduce latency (70ms vs. 200ms for standard consumers)
      - Data over HTTP/2
    - - Trigger notifications / send emails
      - Lightweight ETL - S3, DynamoDB, Redshift, ES ..
    - - Library runs on EC2
      - Firehose replaces for few targets, Lambda for others
      - Write data to: S3, DynamoDB, RedShift, ES
    - - Leverages DynamoDB for coordination and checkpointing
      - Read records produced by KPL
      - Uses a checkpointer to keep track of records that have been processed.
    - - 5 GetRecords API calls / shard / second
      - 2MB aggregate throughput
- - - - Zookeeper
      - Brokers
  - - - S3
      - Cloudwatch
      - Kinesis Data Firehose
- - - - Client encrypt & decrypt data before sending or retrieving from S3
      - Customer manages keys & encryption cycle
      - Client library such as S3 encryption client
    - - SSE-S3
        
        Object encrypted server side
        
        AES-256
      - SSE-KMS
        
        Object encrypted server side
        
        KMS advantage - user control & audit trail
      - SSE-C (client encryption keys)
        
        HTTPS must be used
        
        Encryption key provided in HTTP headers
        
        Data keys fully managed by customer outside of AWS
        
        Can use CloudHSM
  - - - Send S3 Select filtered dataset to Hadoop
  - - - Bucket policies
        
        force encryption at upload
      - Object ACL (fine grain)
      - Bucket ACL (not common)
- - - - Retrieve as soon as object is written
      - EXCEPT if we did a GET before to see if object existed, eventually consistent
- - - - Vault Lock policy - immutable
        
        WORM policy
        
        Forbid deleting an archive
  - - - Expedited - 1 to 5 mins
      - Standard - 3 to 5 hours
      - Bulk - 5 to 12 hours
      - Minimum storage - 90 days
  - - - Standard - 12 hours
      - Bulk - 48 hours