Please enable JavaScript.

Coggle requires JavaScript to display documents.

AWS SAA (Analytics (Kinesis (Kinesis Services (Kinesis Streams (Producers…

- - - - Kinesis Streams
        
        Producers puts data in Shards
        
        Retention: data is stored from 24 hours (default) to 7 days
        
        then they are taken by consumers
      - Kinesis Firehose
        
        data is not stored but automatically processed (there's no consumers), e.g. by Lambda
      - Kinesis Analytics
    - - Data are stored from 24 hours to 7 days
- - - - Amazon currently doesn't support increasing storage on SQL Server Db instance
    - - Automated backups
        
        It takes a daily db snapshot and stores transaction logs through all day. This allows you to recover a db to any point in time within retention period (it can be from 1 to 35 days)
        
        Enabled by default
        
        backups are taken in a defined time window and during a backup you may experience latency
        
        The backup is stored on automatically created S3 bucket. If your RDS db is 10 GB big then there is created for free S3 10 GB bucket
        
        Restored instance is always a NEW RDS instance with a new endpoint
      - Database snapshot
        
        Unlike automated backups they are stored EVEN AFTER deleting RDS instance
      - Multi - AZ
        
        It allows you to to have exact copy of production db in different Availability Zone. Both dbs are automatically synchronised so if one fails then another one comes live.
        
        if there's a planned mainenance window then RDS will automatically switch and then come live again without administrator's action
        
        Multi-AZ is only for Disaster Recovery not for improving performance (for this you need Read Replicas)
      - Read Replica
        
        you have multiple dbs which are synchronised with each other improving performance (e.g. each instance of a web server is connected with an RDS instance)
        
        The other possible purpose is to take a replica of your production for dev purposes
        
        Supported only by MySQL, PosgreSQL, MariaDB
        
        It requires enabled automatic backup
        
        You can have up to 5 replicas of db
        
        Read replicas of read replicas are possible but may cause latency problems , because of asynchronous algorithm of synchronization
        
        Read Replicas CANNOT have Multi-AZ, HOWEVER you can have read replicas of Multi-AZ source dbs
        
        Each replica have its own DNS endpoint
        
        Read replicas can become normal dbs (with write permission)
        
        You can have read replica in a 2nd region however ONLY for MySQL and MariaDB
    - - 2 copies of your data in each AZ, with minimum of 3 AZs = 6 copies of your data
      - self-healing - automatically scans disks for errors and repairs them
      - Replicas
        
        Aurora replicas (up to 15) - provide automatic failover
        
        MySQL replicas (up to 5) - NOT provide automatic failover
  - - - collection = table
        document = row
        key value pairs = field
      - good for gaming, IoT, etc.
    - - Strongly Consistent Reads
    - - Write throughput (per 10 units); expensive for writes
      - Read throughput (per 50 units); cheap for reads
      - Storage costs (per GB)
  - - - single node = 160 GB
      - you can use multi-node
        
        Leader node (receives queries)
        
        compute nodes (store and data and compute queries)
        
        can be up to 128 compute nodes
    - - uses Columnar Data Storage - processing only columns which are involved a query
        
        good for aggregated queries
      - Advanced Compression - Columnar Data Can be compressed more effectively than row based data (it stores same type of data)
      - Massively Parallel Processing - automatically distributes data across all nodes
    - - is based on Compute Node Hours = 1 unit per node per hour
      - charged only for compute nodes
      - you are charged for backup
      - you are charged for data transfer within VPC
    - - in transit SSL
      - at rest AES 256
        
        by default Redshift takes care of key management (but you can change it to AWS KMS or manage keys through HSM)
  - - - Memcached
      - Redis
  - - - E.g. pull up a row where order_id=159
    - - E.g. you want to count net income of EMEA and US region, then you need to have sum of :sold products from EMEA and US, unit cost of each product, sales price of each product etc.
- - - - CPU related,
        Disk related,
        Network related,
        status related
      - Standard monitoring = 5 minutes
      - Detailed monitoring = 1 minute
- - - - Federation - combining list of users in 1 domain (e.g. IAM users) with list of users from 2 domain (e.g. AD)
      - Identity Broker - a service to take an identity from point A and join it (federate it) to point B
      - Identity store - service like AD
      - Identities - users of a service, like AD account
- - - - Retention: SQS has a retention 14 days and SWF up to 1 year
      - Orientation: SQS has a message-oriented API while SWF has task-oriented API (e.g. human involving tasks)
      - Duplicates: SWF ensures that task is NEVER duplicated, while duplicates in SQS are possible
      - tracking: SWF tracks all tasks and events in application, while in SQS you have to implement your own application-level tracking
    - - Workflow starters - an app that can start a workflow
      - Deciders - control the flow ( if the task is finished or fails they decide what to do next
      - Activity workers - carry out the activity tasks
- - - - 2) File Gateway (NFS)
        
        nothing is stored locally
      - Volumes Gateway (iSCSI) - the block based storage (it's like virtual hard drive - you can install on it applications). Data is stored on volumes as Amazon EBS snapshots (limited in size 1 GB-16 TB).
        
        3) Stored Volumes
        
        you're doing a full copy locally and then it is asynchronously backed up (do a EBS snapshot and store it in S3).
        
        4) Cached Volumes
        
        entire data set is stored in S3 and the most frequently accessed data is cached locally
      - 1) Gateway Virtual Tape Library (VTL)
        
        allows you storing your tape data to virtual cartridges
  - - - each object (flat file) is limited to 5TB
        
        each object is build from: key (name), value (data), ID (important for versioning), metadata (e.g. date of uploading) and subresources
        
        subresources contain: ACLs and torrent (it supports BitTorrent protocol)
      - Storage space is unlimited
      - read after write consistency for PUTS of new objects (objects are accessible immediately after uploading)
      - Eventual consistency for overwrite PUTS and DELETES (after modifying or deleting file the changes can take some time to propagate)
      - Storage Tiers
        
        availability = 99,99%, durability = 99,999999999%
        
        data survives even if 2 concurrent facilities will be down
        
        S3 IA (Infrequently Accessed)
        
        lower fee than S3
        
        availability = 99,9%, durability = 99,999999999%
        
        minimum object size is 128KB
        
        what means that smaller objects will be charged as 128KB
        
        Reduced Redundancy Storage
        
        availability = 99,99%, durability = 99,99%
        
        cheaper than S3
        
        Glacier
        
        cheap but takes 3-5 hours to retrieve data
      - Versioning - once you enable it you cannot disable it (you can only suspend it)
        
        Careful with big files, because all versions are stored, what can take a lot of your storage space
        
        It may be used as a backup tool - once you deleted the object, you can steal restore it by deleting the "delete marker"
      - Cross-Region Replication replicates only NEW objects
        
        objects are replicated with their permissions
        
        If you delete an object then the "delete marker" is replicated, however if you delete the "delete marker" in your source bucket (restore the object ) then "delete marker" is not replicated (in your destination bucket the object is still deleted)
        
        When you restore previous version (delete latest version) in your source bucket then it is NOT replicated (in destination bucket it is still in Latest version!)
        
        versioning has to be enabled on both source and destination buckets
        
        you cannot replicate to multiple buckets
    - - each bucket has unique name
      - each bucket can be reached using link: https://s3-eu-west-1.amazonaws.com/[bucketname] or: https://[bucketname].s3.amazonaws.com
    - - actions on current versions
      - actions on previous versions
      - when object expires it means there is removed the delete marker (however you're still able to restore the object)
    - - In-transit - SSL/TLS
      - at rest
        
        Server Side Encryption
        
        S3 managed keys = SSE-S3
        
        AWS KMS - Key Management Service
        
        logs information who is encrypting/decrypting what
        
        uses another key to encrypt your encryption key
        
        Server side encryption with customer provided keys - SSE-C
        
        Client side encryption
    - - it uses the CloudFront Edge Network
      - to upload you're using dedicated URL, e.g. rzepsky.s3-accelerate.amazonaws.com
    - - for some regions the '--region' parameter has to be explicitly specified and for some not. It is better than to use always this parameter so it will always work
      - you can copy whole folder either using 'cp --recursive' or 'sync'. Sync copies ONLY new files. If you want to also include removing files from the destination then use 'sync --delete'
- - - - the event can be for example new file in S3 or an HTTP request (send to API Gateway and then it triggers Lambda function)
        
        each request deploys a new Lambda function
  - - - on Demand
        
        pay fixed rate by hour (or on seconds for Linux)
      - Reserved
        
        capacity reservation. Requires signing a contract for 1-3 years
        
        it is possible to transfer a reserved instance from one AZ to another
      - Spot
        
        you need to set a bid price (a maximum price that you can spend for an hour/second of using EC2). If there is high demand for EC2 (a lot of people is buying it at the moment) then the price of it is going up. If this dynamic price is above your bind your instances will be stopped or terminated.
        
        If you terminate the instance then you're going to pay for this hour. If the AWS terminates your instance then it is for free
      - Dedicated hosts
        
        It may be helpful if the licensing agreement requires it (e.g. an Oracle db) or for the government
        
        can be bought as on Demand or as Reserved
    - - EBS types
        
        GP2 - General Purpose SSD. Balanced price and performance.
        
        3 IOPS per GB up to 10000 IOPS
        
        IO1 - Provisioned IOPS SSD
        
        more than 10000 IOPS
        
        to maximize IOPS performance the best strategy is to add multiple additional volumes with provisioned IOPS and then create a RAID0 stripe across those volumes
        
        ST1 - Throughput Optimized HDD
        
        for large amount sequential data (data warehouse/log processing/Big Data
        
        SC1 - Cold HDD
        
        Lowest cost storage for infrequently accessed workloads (e.g. as a file server,
        
        Magnetic standard
        
        Lowest price per GB
        
        You cannot mount 1 EBS to 2 EC2. Use EFS instead
        
        Only SSD and Magnetic disks can be bootable. The HDD ones CANNOT be root disks (but can be mounted additionally)
      - RAID = Redundant Array of Independent Disks
        
        RAID 0 - striped, no redundancy
        
        RAID 1 - Mirrored
        
        RAID 5 - at least 3 disks, good for reads and bad for writes; not recommended by AWS
        
        RAID 10 - combination of RAID 0 and 1; good performance and redundancy
        
        from multiple EBS volumes you can create a RAID (the EBSes can be of different types), e.g. from 4 different volumes create one striped partition D://
        
        Taking a snapshot of RAID
        
        normally taking a snapshot excludes application and OS cached data, however in RAID it can be a problem due to interdependencies of the array
        
        To take a snapshot of RAID array you have to stop app from writing and flush all caches to the disk
        
        You can do this using one of the following method: freeze filesystem, unmount the RAID array, shut down associated EC2 instance
    - - For each availability zone there is a separated subnet (1 subnet = 1 availability zone).
      - by default the EC2 and its disks are deleted when terminated (but you can change this behaviour)
      - remember about the tagging - thanks to tags you're able to track particular services which generate the costs.
    - - AMI = Amazon Machine Image
    - - 1) Stop an instance and create a snapshot
        2) Copy a snapshot to different region and enable encryption
        2) Create an image (AMI) from this snapshot
      - Snapshots of encrypted volumes are encrypted automatically
      - You can share snapshots only when they are NOT encrypted
    - - The Storage for the Root Device (Root Device Volume) can use either EBS (most common) or instance store (you CANNOT stop it - only reboot or terminate)
      - Rebooting EBS or instance store backed AMI will NOT loose your data
      - Terminating deletes EBS or instance store volumes HOWEVER with EBS volumes you can tell AWS to keep the root device volume
      - Instance store - the root device for an instance launched from the AMI is an instance store volume created from a template stored in Amazon S3 (slower and uses ephemeral storage)
      - Amazon EBS - the root device for an instance launched from the AMI is an Amazon EBS volume created from an Amazon EBS snapshot (they are faster and uses persistent storage)
      - You cannot remove snapshot of an EBS volume which is used as a root device for AMI - you have to firstly deregistered AMI and then you can remove the root device
    - - 2 types
        
        Classic Load Balancer (works in layer 4)
        
        previous generation - forget about it
        
        Application load balancer (works in layer 7)
        
        Successor of previous classic load balancer
        
        Network Load Balancer
        
        Rare choice, only if you need static IP and ultra high performance
      - Require to configure health checks - LB passes traffic only to instances which pass the healthcheck
    - - accessible under the address: http://169.254.169.254/latest/meta-data/
    - - in the auto scaling group you define the amount of instances and the subnets (each subnet is a separate availability zone) - the more subnets the bigger redundancy you have.
        
        In advanced settings you can specify the load balancer which does a health check
        
        Grace period is a length of time before Auto Scaling does a health check
      - allows you specify increase and decrease group sizes - e.g. when the CPU > 90% add an instance and when CPU < 40% remove one instance
      - Auto Scaling automatically re-provision instances, e.g. when you terminate one of your instance it will be automatically (after a while) bring back
      - removing Auto Scaling Group also removes the instances specified in Launch Configuration
    - - It's a logical grouping of instances within a single Availability Zone. It is recommended for applications that need very low latency and very high network throughput (10Gb/s), e.g. clusters.
      - Placement Group cannot span across multiple Availability Zones.
      - only certain instances can be launched in a Placement Group.
      - You cannot merge Placement Groups
      - You cannot move existing instance into Placement Group
        
        However you can create an AMI from existing instance and then launch a new instance from AMI into a Placement Group
      - The name of Placement Group must be unique
    - - PV - Para Virtualization
      - HVM - Hardware Virtual Machine
- - - - Simple
        
        default one
        
        good if you have single resource, e.g. 1 web server
      - Weighted
        
        e.g. 20% goes to server 1 and 80% to server 2
      - Latency
        
        allows you to route traffic based on the lowest network latency
      - Failover
        
        detects a failover and redirects traffic to secondary server
        
        requires creation of health check
      - Geolocation
        
        based on the geographic location (continents, countries or only in US states)
        
        it's helpful when you want for example present one version of a website for all Europeans, and the other version for all Africans
  - - - it isn't only for reading content. You can write to it also.
      - Objects are cached for TTL
        
        TTL is always in seconds
        
        you set it up using "Default TTL"
      - you can cache a web content or streaming RTMP
      - when you PUT a new object to edge location then the edge location it will update your bucket
      - using "Path Pattern" you can set the regular expression what should be cached, e.g. *.pdf
      - You can set up multiple restrictions
        
        WAF
        
        you can whitelist/blacklist Geo-Restrictions
        
        Pre-signed URLs and Pre-signed cookies
        
        you may use pre-signed URLs in the following scenario. You have a website with photos (stored on s3) and you don't want to share directly the photos but rather you want people to visit your website and see the pictures there (e.g. because of displaying ads to people). With normal s3 storage other websites can directly link to photos without reaching your website. If you use pre-signed urls then it is impossible to directly refer to photo. People can only see it on your website.
    - - the origin actually may be your non AWS server - CloudFront will still work
    - - You can delete the distribution by firstly disabling it and then deleting
      - the distribution name is: [random string].cloudfront.net
  - - - There's NO transmitive peering - VPCs can communicate only with directly connected VPCs
      - you can connect VPCs owned by different accounts however only in the same region
    - - Security Groups are stateful, but ACLs are stateless
      - Subnets
        
        1 subnet = 1 availability zone
        
        first 4 IPs and the last from the subnet (5 in all) cannot be used
        
        by default each newly created subnet is asociated with the main route table
        
        by default newly created subnets has no auto-assign public IP - you have to turn it on manually if you want to access resources in this subnet from public
      - Security groups dont't span across VPCs (if you created a security group in VPC1 it will not be visible in ~VPC2)
    - - NAT Gateway - for IPv4
        
        You should have 1 gateway per availability zone
        
        NAT gateway is not associated with security groups (while NAT instances are)
      - Egress Only Internet Access - for IPv6
      - You can use EC2 instance as a NAT GW too.
        
        If you are bottlenecking, increase the instance size
    - - created by default (allows all in and out traffic). But if you create your own one it has a rule DENY everything.
      - you can associate a subnet to only ONE Network ACL
      - Remember to specify a 'Custom TCP' outbound rule with a port range (it is required for ephemeral ports - the ports open on client side, e.g. on client side you're using 1050 as ephemeral port and 80 as a destination port)
      - you can block addresses using ACLs (NOT via Security Groups)
    - - requires at least 2 public subnets
    - - capture logs about the traffic
        
        those logs can be viewed in CloudWatch Logs
      - Can be created on 3 levels
        
        VPC
        
        Subnet
        
        Network Interface Level
      - You can stream captured logs e.g. to Lambda and react for the traffic
      - Flow logs between VPCs are ONLY possible if both VPCs are under ONE account
      - you cannot tag a flow log
      - After creating a flow log you cannot change its configuration, e.g. you cannot assign a new IAM role
      - Not all traffic is monitored. Exceptions are: Amazon DNS traffic (however if you have your own DNS server then it is logged), traffic from Windows instance for an Amazon Windows license activation, traffic to and from 169.254.169.254, DHCP traffic, traffic to reserved IP for the default VPC router
    - - 2 types
        
        Elastic Network Interface
        
        Gateway
        
        works as a NAT gateway
      - accessing any AWS Service through VPC endpoint within one region is done using private IPs
      - they're used for accessing other AWS services behind the gateway (like to the different department, but in the same enterprise network)
- - - - it's like google analytics for mobile apps. Helps you understand user's behaviour.
- - - - (normal) Snowball
        
        80 TB disk space
      - Snowball Edge
        
        100 TB disk space
        
        gives you also the compute capabilities. For example, the airplane engineer takes the snowball edge's box on the board and it's mounted as a disk. During the flight data about engines is collected and then send to Amzaon data centre. In the result you have in your cloud not only the data, but also the Lambda function.
        
        Looks the same as normal snowball
      - Snomobile
        
        up to 100 PB
    - - you have to download the manifest file and provide the access code from your AWS console to run the Snowball
- - - - 2 types of queues
        
        standard (default)
        
        generally works as fifo but occasionally some messages can be delivered out of order (but the deliver is guaranteed)
        
        FIFO queues
        
        guarantees the order and deliver
        
        limited to 300 transactions per second
      - messages can be kept in a queue from 1 minute to 14 days (default is 4 days)
      - long polling doesn't returns a response until a new message appears in queue (in opposite to short polling)
- - - - Docker container are run based on JSON Task definitions