Please enable JavaScript.

Coggle requires JavaScript to display documents.

GCP - Cloud Architect, App engine, Access to services via API - Coggle…

- - - - Scopes are in URL format googleapis.com.auth/bigquery.insertdata
      - After changing the scope - instance must be restarted
      - Owner on ORG level is owner for all projects but not real owner for ORG.
  - - - Resource based
        ex:
        
        resource.type=="compute.googleapis.com/instance"
        
        resource.name.starts.with("machine")
        
        resource.service=="storage.googleapis.com"
      - Request based
        ex:
        request.time<timestamp("2021-01-01T00:00:00Z")
        source IP....
- - - - between regions (or zones which do not support auto-move):
        1) create source disk snapshot
        2) create new disk from snapshoot in destined zone
        2.5) new image? Check it!
        3) create new machine from new disk in destined zone
        4) assign new subnet
        5) assign new disk
        6 ) delete old snapshot, disk and vm instance
        between zones: public ip address can be persisted (change ephemeral ip to static and reassign to new machine)
        between regions: new public ip address must be assigned
      - between zones - using API
        gcloud compute instances move VM --zone X --destination-zone Y
    - - Persistent disk
        block storage via network
        
        Zonal persistent (standard or SSD)
        
        Regional persistent (standard or SSD) - redundancy - replicated to another zone
      - Local SSD - very efficient (physically attached to host) - data is not persistent, lost when VM is stopped/terminated (interface types: SCSI or NVME)
      - Efficiency increases with more CPU and disk space
      - gcloud compute instances attach-disk....
        
        sudo mkdir -p /disk2
        sudo mount -o discard, defaults /dev/sdb /disk2
        sudo chmod a+w /disk2
  - - - bases on:
        
        existing persistent disk
        
        other image
        
        snapshot
        
        RAW image in Cloud Storage
  - - - upgrade to containers GKE
    - - Migrate for Compute Engine (options) - stream data to cloud
        
        test clone in GCP (with real data on prem) - VM on prem is running
        
        run in cloud (streams storage from on prem as needed) - local VM is stopped
        
        Full migration
      - import disks/images to GS storage - no modification/ 1:1 - fastest
- - - - Deployment -> Staging -> Production
      - NodePool - group of nodes with the same configuration
        Custom node Pool - nodes can have different config (but cannot autoscale)
      - Release channels: Rapid / Regular / Stable
      - GKE management models
        
        Imperative - manually, by commands
        
        Declarative - by templates (easier to rollback)
    - - Authenticate:gcloud container clusters get-credentials [CLUSTER-NAME] --zone=us-central1-c
        
        create POD: kubectl apply -f pod.yaml
        
        Apply updated PODs config: kubectl apply -f nginx-deployment.yaml
        
        Deployment: a set of PODs (replicas) in cluster. View deployments: kubectl get deployments
        
        kubectl get endpoints nginx-deployment
        
        kubectl create deployment hello-server --image=gcr.io/google-samples/hello-app:1.0
        
        usuallly: 1 POD = 1 container
        but can be: 1 POD = N containers
        
        kubectl get pods
        
        Get PODs with particular labels: kubectl get pods -l "app=nginx"
        
        Label PODS
        kubectl label pods secure-monolith 'secure=enabled'
        
        kubectl get pods secure-monolith --show-labels
        
        kubectl get nodes (get NODES / WORKERS
        
        Publish PODs via Service
        kubectl expose deploymentx enginx --port=80 --type=LoadBalancer (network lb)
        
        Delete POD: kubectl delete pod -l name=nginx
        
        kubectl get service hello-server
        
        Execute command in container:
        kubectl exec monolith --stdin --tty -c monolith /bin/sh
        
        Containers in POD communicate using LOCALHOST / fixed ports
    - - Container mechanisms:
        
        Control groups - manages container resources (CPU, mem)
        
        Kernel Namespaces - isolates processes from other containers or HOST
        
        Union filesystem - FS
      - Dockerfile - file specyfing steps to create Docker image
        image is immutable
      - Basic objects
        -specs and status
        
        POD - basic kuberenets unit - include containers
        
        Namespaces - virtual environment
        
        Service - exposes application
        
        ReplicaSet - guarantee the availability of identical PODs replicas
        
        DeamonSet: Ensures that POD copy is runnign on each NODEs, even new ones. We use it when we want to do some operations on all workers/nodes - gathering logs and monitor applications.
        
        StatefulSet - each POD replica has unique hostname + index, so we can identify them.
    - - Architecture: master+ N nodes (workers):
        
        master
        
        controller manager - manages nodes/workers and PODs
        
        etcd - configuration
        
        api server - modify cluster
        
        scheduler - manages new PODs
        
        N nodes (workers) to run containers
        
        kubelet - agent on each NODE, ensures that PODs functions properly
        
        kube proxy - services
        
        All components talk to API server. API servert talks to ETCd
    - - Manual
        kubectl scale deployment nginx --replicas=3
      - HorizonatlPODAutoscaling HPA
        
        kubectl autoscale deployment nginx --cpu-percent=50 --min=1 --max=10
        
        kubectl get hpa nginx
- - - - The primary IP range for the subnet can be expanded but not replaced or shrunk after a subnet has been created
      - Subnets: min /29
        max /16
      - Secondary range can be removed if not used
    - - VPC Flow Logs
        
        monitor traffic between instances
        
        ingress /egress traffic to VPC
        
        written to StackDriver
        - enabled per subnet
        
        Aggregation - packets packed into single flow / entry
        
        Flow Log sampling (every 10th packet is sampled)
        
        Metadata - (geographical location)
        
        Writing to StackDriver
      - Firewall logging
        
        logging to: Stackdriver / Pub/Sub / BigQuery
        
        logs each connection (not sampled)
        
        only TCP/UDP traffic
  - - - Private Google Access
        
        allows instances without public IPs yo access Google APIs
        
        enabled per subnet
        
        Private Google Access for on-premises hosts
        
        to Google APIs for on premise hosts (requires Cloud VPN or Cloud Interconnect)
      - Dedicated Interconnect
        
        Direct access to Google network
        
        10 or 100 Gbit/s
        
        max 8 x10 Gbit /s or 2 x 100 Gbit/s
        
        we encrypt by ourselves
        
        SLA
      - Partner Interconnect
        
        Access through supported provider
        
        50 Mbit to 50 Gbit/s
        
        L2 (BGP on prem router direct to GCP cloud router) or L3 (BGP partner router to GCP cloud router)
        
        SLA
    - - Cloud VPN
        
        only GCP resources
        
        we encrypt traffic (IPSEC)
        
        2 options: HA VPN (2 tunnels to 2 public IPs -active:active or active:standby) or classic VPN (single public IP, but can have 2 tunnels)
        
        on prem router must support BGP
        - up to 3 Gbit/s
        
        GCP provides SLA 99,99% on their side
      - Direct Peering
        
        via Cloud VPN
        
        free of charge
        
        no SLA
        - up to 10 Gbit/s per link
        
        Google location must be close to us
      - Carrier Peering
        
        via ISP
        
        no SLA
        
        costs of Service Provider
  - - - In Premium Tier: Backends can be in any region and any VPC network.
        In Standard Tier: Backends must be in the same region as the forwarding rule and lb IP, but can be in any VPC network.
        
        External SSL Proxy
        
        IPv4 and IPv6
        
        global or regional
        
        for premium network only
        
        intelligent routing to closest region
        
        target proxy
        
        certificate management (self or by Google)
        
        TCP Proxy Load Balancer (for non http traffic)*
        
        intelligent routing (global)
        
        ipv4 and ipv6
        
        target proxy
        
        forwarding rule
        
        backend services or buckets
        
        established 2 sessions
        1: ext client -> LB public IP
        2: LB -> to backend
        
        External HTTP(s) Load Balancer
        
        supports IPv4 and IPv6
        
        connections from outside is termninated at LB. New connection is made to backends
        
        forwarding rule
        
        target proxy (many GoogleFrontEnds)
        
        URL map
        
        backend service
        
        backends can be instance groups or buckets
        
        SSL requires cerificate installation (up to 15)
        
        supports Google Cloud Armor
        
        established 2 sessions
        1: ext client -> LB public IP
        2: LB -> to backend
        
        steps to create: 1) instance-groups 2) healthcheck, 3) named ports 4) backend 5) add to backend 6) ip 7) url-map 8) http proxy 9) forwarding-rule
      - External TCP/UDP network Load Balancer
        
        regional only loadbalancing
        
        send traffic directly to instance (not proxy, not terminates)
        
        backends are called target pools
        
        healthchecks
        
        Hashing algoriths (lb algorithm)
        
        client IP
        
        client IP + protocol
        
        client ip + protocol + port
        
        none
    - - HTTP(s) LoadBalancer
        
        layer 7
        
        internal IP
        
        forwarding rule
        
        backend service
        
        URL map
        
        Healthcheck
        
        target proxy
        
        Proxy only subnet - Envoy (traffic to backends)
        
        for Compute Engine/Kubernetes Engine
        
        premium network only
        
        Limitations
        
        must be placed in chosen region
        
        request may be handled in different zone
        
        does not support IAP, Cloud CDN, buckets, VPC peering, WebSocket
        
        clients and backends must be in the same region
        
        don't support backend buckets
      - TCP/UDP Load Balancer
        
        backends must be in the same region and VPC
        
        is not accessible from internet
        
        can be accessed via VPN and VPC peering
        
        send traffic directly to instance (not proxy, not terminates)
        
        internal IP
        
        forwarding rule
        
        backend service
        
        healthcheck
        
        premium network only
        
        Hashing algoriths (lb algorithm)
        
        client IP
        
        client IP + protocol
        
        client ip + protocol + port
        
        none
- - - - Backups
        
        Automated (in 4 hour windows) or On-demand
        
        Restore - if restored db has read replicas, they need to be deleted and recreated after restore
        
        Point in Time - within 7 days (for automated with binary logging enabled). Binary logging - consumes extra paid space + degrades writes
      - High Avaialbility
        
        regional (2 different zones)
        
        single IP address
        
        persistent disks are replicated
        
        master (primary) and standby reads from the same primary disk.
        
        standby becomes master when there's no keepalives from master within 60 s
        
        add more disks to increase performance
        
        to move to another region: create replica and make standalone (master)
  - - - Cloud Bigtable(NoSQL wide column)
        
        HBase - based (Petabytes+)
        
        global | regional
        
        horizontal scaling
        
        multi primary
        
        low latency, high IO
        
        replica consistency: eventual
        
        data stored on shared storage - Colossus
        
        data serving NODES can be easily and fast replaced (only metadata / pointers replicated)
        
        data matrix as key/value
        
        best for: time-series data, historical, telemetric data, analytical, IOT, finance
      - Cloud Firestore - collections(old Datastore - entities)
        
        NoSQL, schema might change
        
        (Terabytes+), can scale DOWN to ZERO
        
        document based (JSON files)
        
        regional | global
        
        multi primary
        
        serverless
        
        not very fast
        
        no relations on schema level
        
        Datastore - for server apps
        
        Firestore - for Mobile/Web apps (user profiles, game states)
        
        no strict columns - only type must match
        
        Data export: Firestone => Cloud Storage => Cloud Scheduler => Pub/Sub => Clud Functions (process/ get data) => Cloud Storage
        
        INDEXES: 1) included (index on each filed) 2) composite (we define indexed fields)
        gcloud datastore index create
    - - Availability:
        
        BASIC: connections lost when scaling
        
        STANDARD: extra replica created during scaling (with HA / multizone replica)
        
        master / replica - uses the same IP
        
        can serve services in the same region only
  - - - Buckets: Lifecycle Management
        
        Delete
        keep versions
        
        Retention: keep data for amount of time (cannot be changed)
        
        Set storage Class
        
        Storages classes:
        
        Standard (frequent access)
        
        Nearline (once a month, for backups, at least 30 days)
        
        Coldline (once a year, archive, DR, min 90 days)
        
        Archive
        
        Each class can have availability type:
        
        Multiregional
        
        Dual Regional (global, frequent access data). Replicated, GEO redundant
        
        Regional (Regional)
        
        gsutil mb -c standard -l us-central1 gs://testbucket001
        
        Data in buckets have data +metadata (ACL, cache, presentation, type of object, etc or custom)
        
        bucket name can have slash /
        
        infitnite amount of objects in bucket
        Objects are immutable, can be replaced, not changed
        
        max 5 TB single object
        
        creation / changes / delete or archive can be integrated with PUB/SUB to inform
        
        Composite objects (created from many objects):
        
        must be in the same bucket
        
        cannot be encrypted
        
        must have the same storage class
        
        GSUTIL
        
        Parallel copy of local files to Cloud Storage bucket.
        gsutil -m cp gs://gcp-bucket-nr1*
        
        Display total size of all objects from a bucket.
        gsutil du -s gs://gcp-bucket-nr1
        
        View metadata information about a bucket.
        gsutil ls -L -b gs://gcp-bucket-nr1
        
        Creating another bucket with "standard" Storage Class and in "us-east1" region.
        gsutil mb -c standard -l us-east1 gs://gcp-bucket-nr2
        
        gsutil rm -f -- removes objects
        
        Copy all files from "gcp-bucket-n1" to "gcp-bucket-nr2" (not folder structure).
        gsutil cp gs://gcp-bucket-n1/** gs://gcp-bucket-nr2
        
        List all data from bucket-nr2.
        gsutil ls -r gs://gcp-bucket-nr2/**
        
        Change default Storage Class of "gcp-bucket-nr2" to "nearline".
        gsutil defstorageclass set nearline gs://gcp-bucket-nr2
        
        Setting retention on bucket -object must exist for a given time, cannot be deleted
        gsutil retention set 60s gs://bucket-991
        - Set ACL on bucket object
        gsutil acl ch -g AllUsers:R gs://gcp-bucket-nr1/img3.jpg
        
        Set ACL via IAM:
        gsutil iam ch user:john.doe@example.com:objectCreator gs://ex-bucket
        
        Logging ON
        gsutil logging set on -b <logging_bucket_name> [-o <log_object_prefix>] gs://<bucket_name>
        
        other options: rsync (gsutil rsync data gs://bucket/data) / setmeta / web / iam / config
        
        copying stream: long_computation | gsutil cp - gs://bucket01
        
        Paralel Composite Uploads: gsutil -o GSUtil:paralel_composite_uploads_threshold=150M cp bigfile gs://bucket01
        
        Signed URL
        Access to URL object int the context of service account (as proxy)
        gsutil signurl -d 10m key.json gs://gcp-bucket-nr1/img1.jpg
        
        Versioning
        
        generation (data version)
        
        metageneration (metadata version)
        gsutil versioning set on gs://gsbucket01
        gsutil versioning get gs://gsbucket01
        gsutil ls -a gs://gsbucket01 --show archive ver
        
        Policy - when multiple rules, always most cost effective wins. Policy specified in JSON file:
        
        gsutil lifecycle set policy1.json gs://gcpbucket01
        
        gsutil lifecycle get gs://gcpbucket01
        
        Criteria
        
        Conditions:
        
        Age
        
        CreatedBefore
        
        IsLive (TRUE for latest version)
        
        MatchesStorageClass
        
        Actions:
        
        Delete
        
        SetStorageClass
      - Data transfer services
        
        gsutil cp -m
        
        Storage Transfer appliance - Petabytes (on premise) (>10TB, > 1 week to upload, service is avalable in your location)
        
        Storage Transfer service (> 1TB and < 10 TB) - HTTPS (from other cloud buckets, other cloud providers - ex AWS) - task once or as a scheduled job
        
        Big Query Transfer Service
- - - - Project
        
        ID and Number (immutable)
        
        Display Name (mutable)
        
        Lifecycle state
        
        Labels
        
        Creation time
- - - - LoadBalancing (L3
        
        L3 - TargetPool - network LB, within region
        
        L7 - BackendService - service NamedPorts
      - Autoscaling
        
        CPU utilisation
        
        HTTP LB utilization | request /second
        
        Stackdriver metrics (qty users online)
    - - Regional MIG - multizone
      - Autohealing - based on MIG healthcheck
        REMEMBER to allow healthcheck traffic from outside to instances (http/https)
      - Zonal MIG - single zone
    - - Made by Updater API
        
        MaxSurge (extra created instances for update)
        
        Max Unavailable - max instances offline during update
        
        MinReadySeconds - amount of time after update for instance to consider is healthy
        
        TargetSize - how many instances will be updated
        
        MinimalActions
        
        NONE - do not interfere instance work
        
        REFRESH - does not stop instance, perform updates
        
        RESTART - stop/start instance, update OS
        
        REPLACE - recreate instances
      - gcloud compute instance-groups managed rolling-action start-update my-group
    - - Loadbalancer: avoids unhealthy instances, directs traffic toward healthy
      - MIG healthcheck- recreates unhealhty instances
- - - - Architecture
        DREMEL (compute engine):
        Root Server (gets query) => Mixers (optimize query) => Leaf nodes. Results are sent back to Mixers => Root Server
        COLOSSUS (Data storage)
      - Common commands:
        
        bq mk Dataset1 - create dataset
        
        bq ls - list datasets
        
        bq show - display dataset info
        
        bq load - load data and create table
        
        **bq query - use_legacy_sql=false \
        'SELECT
        word,
        COUNT(names) FROM
        bigquery-public-data.samples.shakespeare
        WHERE
        word LIKE "%raisin%"
        GROUP BY
        word'**
  - - - Design
        
        Stateless functions
      - Deploy
        
        inline console editor
        
        zip upload
        
        zip upload from Cloud Storage
        
        Cloud Source Repository