Please enable JavaScript.

Coggle requires JavaScript to display documents.

System design, CDN- Content Delivery Networks, Microservices, EVENT DRIVEN…

- - - - High level desing diagram
        
        Load balancer -> servises ->DB
      - Data model and schema
        
        data access paterns
        
        read/write ratio
      - APIs - contract between user and backend
        
        follow RESTful convention
        
        check requests and responses
      - Review all high level design
  - - - come up with at least 2 solutions
      - discuss the trade-offs of the solutions
      - articulate the problem
      - pick the solution and discuss it with interwier
      - repeat for all problem
  - - - consistensy
      - scalability
        
        Scalability
        
        user-><-server - latensy, CPU
        
        Verticcal scaling
        
        increas processing power of single server
        
        pros
        
        data consistency
        
        fast inter-proccess cominications
        
        corn
        
        single point of failure
        
        economic(hardware) limits of scaling
        
        Horizontal scaling
        
        host the app in more servers that run in parallel
        
        pros
        
        more reliable
        
        scales lineary
        
        corns
        
        data inconsistancy
        
        difficult to manage
        
        RPC communications (network communication) - slower
        
        need load balancer
      - security
      - performance
      - freshness
      - availability
        
        HI availability = 99,999% = 5 min of downtime per year
        
        Availability paterns
        
        failover
        
        db & servers
        
        how we shift from failing component to others
        
        active-passive patern
        
        there are 2 servers active(recives requests) and passive(sleep and liasten heartbeets of active), one active is fall, passive start works - it's increase latency
        
        active - active patern
        
        there is load balancer and both servers are managing traffic, no downtimes - if one fall another is working
        
        replication
        
        relatives DB
        
        paterns
        
        primary & replica
        
        split read and write requests - write to one primary DB and read from replicas
        
        2 more items...
        
        primary & primary
        
        for each server his own primary DB for read/write(which are replicas for each other), servises are scale horizontaly
        
        2 more items...
- - - - depends on
        
        disaster type
        
        geografical impact
      - BCP
        
        business continuity plan - set of actions when disaster
    - - impact by
        
        network issues
        
        load spikes
        
        CDN допомаг зниз
        
        components failure
- - - - SPLIT BRAIN
        
        Solve
        
        add 3 DB - when connection between A and B is broken ->write operation to A -> path update to B -> cant connect-> path update to C-> C is updated write operation to B->path update to A->cant conect->path update to C -> compare previous state of B with own -> they not same-> B roll back transaction-> update state to C state -> write new data and only than C take update from B....
- - - - data format - key-value pair
      - store data in a memory as a log file
      - periodicly this data damp in sst table(sorted string table) - ley is sorted
      - sst is immutable
    - - update will create copy of the record with timestamp - if you want to read - need take data with latest timestamp
        
        to deal with that
        
        COMPACTION
        
        splash SST to save space
    - - set flag tumstone(ts)
        
        thats mean thats record is dead
- - - - DB
        
        user profile
        
        MySQL
        
        user metadata
        
        key-vslue
        
        casandra, redis
        
        faster read operation then MySQL
        
        schema flexible
        
        video
        
        amazon s3
        
        gives us
        
        availability
        
        CDN
        
        reliability
        
        file storage
    - - in parallel
        
        split video in two chanks -> convert each chank in different formats(if video short - less than 10sec -> we will not split it)
        
        convert that formats into different resolutions
        
        after that all the chanks will be combined together( result 8-16 copies depends on amount of resolutions)
- - - - faster
  - - - accurate, more resiliant for crashes
      - scale cashe independently without scaling servises
      - all serverse heet global cash if data upsent - heet DB
- - - - new put on top, delete from bottom
- - - - might be inconsistensy better to use write-back or hybrid of both
- - - - set capacity for every server - request per second
        
        if queue gets more requests then capacity -> request falied but others stay in a queue
        
        message that get user with failed request with error
        
        errors
        
        temporary
        
        permanent
- - - - sessions microservisec - prevent single point of failure
        
        DB - save data what use connected to what box
        
        session servise - using TCP - WEBSOCKETS(WSS)
        
        appropriate gateway
        
        user B
        
        allows pair to pair communication
  - - - when userB get mes - it sent notification to session server that mes was got(delivered) -> userA
        
        same for read notification
      - message stores i DB before it delivered to userB
- - - - Last Seen servise
        
        DB - save user id and time stamp of the last activity(in a pair)
  - - - lastseen service
        
        DB
        
        ... userB
- - - - Session server
        
        group server
        
        DB - store users id which belongs to groupId (one to MAny) - using consisting hashing
        
        sent it to sesion server
        
        DB - find number of gateway of every user
        
        send message to users
- - - - file
        
        Gave us
        
        faster - save images separetly, quick access
        
        CDN
        
        cheeper
      - blob - binary large object
      - DB gives us
        
        transaction - ACID
        
        useless for images
        
        indexes for search
        
        useless for images - image saves in format of 0 and 1
        
        mutability
        
        unnesessary for images
        
        access control
        
        useless - you can get same secure mechanism for files as for DB
        
        save parh to image in table but image stores in DFS - distributed file system
      - user -> reqest (update name) need to have autentification process(name, token) -> Gate way service-> profile servise -> DB->...-> response -> user
      - user-> request - update images -> gateway serv-> image servise-> DB (store user id, image id, name, path to DFS) and DFS
  - - - every user has his conection id
      - user-> gateway service-> session servise->DB - store user id and connection id
- - - - high load of DB
    - - inefficient - bad user expirience
- - - - bad user expirience
    - - user get response that their file was uploaded and waiting for the processing
        
        file execute later when server has capacity - asynchronous processing of events
        
        check output file and write results to the DB
  - - - virtualization - create virtual system in which code can run
      - containers - create for every request
- - - - prepare
- - - - +
        
        Post table
        
        id
        
        text
        
        timestamp
        
        user id
        
        image
        
        Comment table
        
        id
        
        text
        
        timestemp
        
        Like table
        
        timestamp
        
        user id
        
        activity id (id of post or comment)
        
        active or delited
        
        Type(comment or post)
        
        id
        
        Activity table
        
        id
        
        likes
    - - -
  - - - followee id
      - timestamp
      - follower id
- - - - internal protocol
        
        UserFeedSeervice - need multiple servers to make system resiliant
        
        PostsServ (multiple)
        
        FolowServ(multiple)
        
        getUsersFollowedByUserId(id)
        
        getPostsByUser(id)
        
        limit post to 20
        
        Better way - PRECOMPUTE feed
        
        when user from list of folowee add new post -> PostService -> UserFeedService-> update feed for user with limit of 20
        
        place to save feed for every user - in DB or cashe
      - Load balancer(use hashing of id to figure out what service needed)
- - - - total uploaded video - 1M*10min = 10M min
        
        total video size = 3 MB * 10M = 30 TB
        
        2 hour video = 4 GB
        
        optimized codac and compretion - 90%
        
        2 hours video optimized size = 4:10 = 0,4 GB
        
        1 min avrg size = 0,4:120 = 3 MB
- - - - 1 NO - mark as suspitious
      - 2 NO - mark as dead
        
        -> Load balancer
        
        ask someone restart server
        
        redirect requests
- - - - uniform assigment
      - scalable granularity - brake regions on smallest regions
- - - - converting to different resolution and formats
        
        need Job Assigner(message queue) - that will create job that convert all videos
        
        assign jobs to different workers
        
        when job is done - sent message with results to the message queue(other)
        
        subscribers will took video from queue
        
        and save new format of video in DFS
        
        send notification to the Transformation service that job job os done
        
        workers pulls another task from first queue