Please enable JavaScript.

Coggle requires JavaScript to display documents.

2022 Oct - 2022 Dec - Coggle Diagram

- - - - because VPC peering its non trnsitive
      - Alternative soolution AWS transit gateway
      - if you do not want to create a mesh you can create VPC peering
  - - - reaching to other services such as SNS, SQS , CloudWatch , eyc
    - - Subscribed to SAAS services
      - When provider wants to privately expose to your AWS resources
        
        They have their own AWS services
      - Through this service you can reach out to exposed application services via a network load balancer set up at provider end
- - - - AWS nowmobile
      - AWS Data Sync
      - Amazon Kinesis data firehose
      - AWS data migration services
    - - Glue
      - Redshift
      - Amazon EMR
    - - Amazon Quicksight
      - Amazon Machine Learning
      - AMazon sage maker
      - AWS market places
    - - RDS
      - S3
  - - - when data grows the performance should not be compromised
      - dynamically scale to meet the requirements
      - when no of users increase system should be able to handle
    - - even with thousands of concurrent users and queries quick results should be available
    - - users should be able to do their own reports without having much support from the data experts
    - - means to handle unstructured data
  - - - batch or real time
      - ability to use data from multiple data sources - structured or unstructured
      - Tools
        
        AWS Data sync
        
        AWS data migration services
        
        AWS snowball
    - - leader node
        
        distribute queriesa across different compute nodes
      - compute nodes
        
        machine which can be partioned and smaller slices
        
        those slices are used to process the queries
        
        symetric multi processing - highly paralla processing
      - HW platforms
        
        HDD - if storage is important
        
        SSD
        
        if latency is important
      - managed data compliance
      - Security
        
        end to end encrypted
        
        AWS KMS integrated
        
        network seggregation
      - Redshift anti patterns
        
        if the data is less than 100 GB
        
        OLTP
        
        Online transaction processing
        
        can use traditional RDBMS or No SQL databases
        
        Unstructured data
        
        data must be first structured by a defined schema
        
        Amazon Redshift spectrum
        
        BLOB data store
        
        USE S3 instead and reference in Amazon Redshift
      - design considerarions
        
        Go for binary options
        
        JSON and YML are more human readable not ideal for Redshift processing
        
        Parquet and Avro and ORC
        
        distribution balancing
        
        better for optimially usage
        
        option - round robin
        
        Option - join - distribution by key
        
        All - full copy on each copy - storage wise expensive
        
        DC 1 or DC2
        
        recommends most recent designs
        
        recent query retrieval
        
        short query accellaration
        
        they can outside outside the long queries
        
        Elastic resize
        
        if you need more power you will get more nodes - auto scaling
        
        concurrency scaling
        
        spins up different clusters outside your main cluster
        
        bursting new clusters
        
        Spectrum
        
        use the advantage of S3 and still use the performance of Redshift
- - - - consumption archetypes
        
        Digital applications
        
        Advanced analytics
        
        must allow AI and ML
        
        Reporting Systems
        
        Involves highly governed data
        
        More quality concious
        
        Security is paramount
        
        discovery sandbox
        
        combination of raw and aggregated data
        
        usually used by data scientists and data engineers
        
        external data sharing
        
        adhere to strringent policies
        
        how data is managed
        
        how it is secured
        
        how data could be consumed
  - - - https://www.youtube.com/watch?v=9Mm5l-l9OPo&list=PL7bE4nSzLSWfya8-VAIM-H40UUOVV2ddZ
    - - when queries are made to work faster then processing time reduces and it will improve saving the costs
    - - think about compressing the files that are not frequently used. This will save space and save costs
    - - Do this based on business requirements.
      - Usually partition by time works
      - If the data is queried based on year then do not go and partition for month
- - - - https://getdx.com/podcast/scaling-infrastructure
- - - - https://hbr.org/2017/11/many-strategies-fail-because-theyre-not-actually-strategies?utm_medium=social&utm_campaign=hbr&utm_source=twitter&tpcc=orgsocial_edit
    - - A real strategy involves a clear set of choices that define what the firm is going to do and what it’s not going to do
      - Many so-called strategies are in fact goals
        
        It does not tell you what you are going to do; all it does is tell you what you hope the outcome will be. But you’ll still need a strategy to achieve it.
      - Others may represent a couple of the firm’s priorities and choices, but they do not form a coherent strategy
    - - As a new strategy, Martin aimed (1) to make perfect scale models (rather than toys); (2) for adult collectors (rather than for children); (3) that appealed to a sense of nostalgia (because it reminded adults of their childhoods)
      - it represented a clear set of just three choices, which fit together to form a clear strategic direction for the company.
    - - easy to communicate
      - Consider Hornby again. Its employees — product designers and technical engineers, for example — could all tell me their company’s new choices.
    - - Successful firms are characterized by maintaining bottom-up internal experimentation and selection processes while simultaneously maintaining top-driven strategic intent
    - - A common mistake in the bottom-up implementation process is that many top managers cannot resist doing the selection themselves.
      - trategy implementation requires top managers to design the company’s internal system that does the selection for them
    - - Finally, another reason many implementation efforts fail is that they usually require changing people’s habits
- - - - enjoy the journey
      - good to have a destiation as a goal
    - - everything will look like a black box until you start opening the box and explore. It will not look complicated
    - - specially in tech space
      - If there is a new product that help me to consume, connect or create
    - - Not having something to having someting
      - Not iphone 8 to iphon 10 which is a upgrade on what you have
    - - Holding you back
    - - Save time
    - - EVeryone is friendly