Please enable JavaScript.

Coggle requires JavaScript to display documents.

Designing Data Intensive Applications, System Design - Coggle Diagram

- - - - Fault vs failure distinction
        
        Fault: Component deviation from spec
        
        Failure: System unable to provide service
      - Redundancy strategies
        
        Hardware redundancy
        
        Disk mirroring (RAID)
        
        Redundant power supplies
        
        Software redundancy
        
        Replication across nodes
        
        Microservice architecture
        
        Human fault tolerance
        
        Automation for error prevention
        
        Clear operational procedures
    - - Failure detection systems
        
        Heartbeats and monitoring
        
        Timeout mechanisms
      - Automated rollback procedures
        
        Transaction rollback mechanisms
        
        State restoration from snapshots
        
        Logging and replay of operations
      - Manual intervention protocols
        
        Escalation paths for critical failures
        
        Runbooks for known failure scenarios
  - - - Horizontal vs vertical scaling
        
        Horizontal scaling
        
        Adding more machines to distribute load
        
        Sharding and partitioning strategies
        
        Vertical scaling
        
        Upgrading resources of a single machine (CPU, RAM)
        
        Limitations due to hardware constraints
      - Elastic resource allocation
        
        Autoscaling in cloud environments
        
        Dynamic resource provisioning
    - - Latency reduction techniques
        
        Caching mechanisms (client-side, server-side)
        
        Query optimization
        
        Connection pooling
      - Throughput maximization
        
        Parallel processing and concurrency
        
        Load balancing across instances
        
        Efficient data serialization
  - - - Monitoring implementations
        
        Metrics collection (CPU, memory, latency)
        
        Logging levels (debug, info, warn, error)
        
        Distributed tracing (request flow analysis)
      - Automated alert systems
        
        Threshold-based alerts
        
        Anomaly detection mechanisms
        
        Notification channels (email, Slack)
    - - Schema migration patterns
        
        Online schema changes (rolling updates)
        
        Offline schema migrations (downtime required)
        
        Blue/green deployments
      - Backward compatibility
        
        API versioning strategies
        
        Data format evolution (e.g., Avro schemas)
        
        Support for older client versions
- - - - Schema-on-read vs schema-on-write
        
        Schema-on-read (Document DBs)
        
        Flexibility for evolving data
        
        Interpretation at query time
        
        Performance overhead during read
        
        Schema-on-write (Relational DBs)
        
        Enforced data consistency
        
        Structure defined at write time
        
        Schema migrations for changes
      - Nested data implementations
        
        JSON document storage examples
        
        MongoDB
        
        Couchbase
        
        XML document storage examples
        
        Legacy systems
        
        Configuration files
    - - Property graph implementations
        
        Nodes and edges with properties
        
        Cypher query language (Neo4j)
        
        Use cases: social networks, recommendation engines
      - RDF/Triple Store applications
        
        Subject-Predicate-Object triples
        
        SPARQL query language
        
        Semantic web and knowledge graphs
        
        Example: Wikidata, schema.org
  - - - SQL optimization engines
        
        Query parsing and validation
        
        Query rewriting and normalization
        
        Index selection strategies
      - Cost-based query planning
        
        Estimating query execution cost
        
        Considering CPU, I/O, network costs
        
        Choosing optimal execution plan
    - - Event sourcing architectures
        
        Immutable event logs
        
        Rebuilding state from events
        
        Audit trails and history tracking
      - Complex event detection
        
        Windowing operations (time-based, count-based)
        
        Pattern matching across event streams
        
        Real-time anomaly detection in streams
- - - - Strong vs eventual consistency
        
        Strong consistency (Linearizability)
        
        All reads see the latest write
        
        Requires coordination and consensus
        
        Higher latency, lower availability
        
        Eventual consistency
        
        Replicas converge over time
        
        Higher availability, lower latency
        
        Potential for read staleness
      - Quorum-based synchronization
        
        Write quorum (W nodes to acknowledge write)
        
        Read quorum (R nodes to read for consistency)
        
        Typically W + R > N (total replicas)
    - - Leader-follower replication
        
        Single-leader replication
        
        One node for writes, others replicate
        
        Simpler consistency management
        
        Single point of failure (leader)
        
        Multi-leader configurations
        
        Multiple nodes accept writes
        
        Conflict resolution needed
        
        Higher write availability, complex consistency
      - Leaderless replication
        
        No dedicated leader node
        
        All nodes can accept reads and writes
        
        DynamoDB, Cassandra
        
        Quorum reads and writes for consistency
  - - - Hash vs range partitioning
        
        Hash partitioning
        
        Distribute based on hash of key
        
        Even data distribution
        
        Range queries inefficient
        
        Range partitioning
        
        Keys divided into contiguous ranges
        
        Efficient range scans
        
        Potential hotspots if keys are not uniform
      - Hotspot mitigation techniques
        
        Salting keys for hash partitioning
        
        Pre-splitting ranges for range partitioning
        
        Dynamic rebalancing of partitions
    - - Local vs global indexes
        
        Document-Partitioned (Local) Indexes
        
        Index on each partition for local data
        
        Queries may need to scan all partitions
        
        Simpler index management
        
        Term-Partitioned (Global) Indexes
        
        Global index across all partitions
        
        Efficient lookups across entire dataset
        
        More complex to maintain and coordinate
      - Multi-dimensional indexing
        
        Spatial indexes (R-trees, Quadtrees)
        
        Geographic data queries
        
        OLAP indexes
        
        Multi-attribute filtering
        
        Example: color, size, price range for product search
- - - - Read committed/Snapshot isolation
        
        Read Committed
        
        Prevents dirty reads (reading uncommitted data)
        
        Non-repeatable reads possible
        
        Snapshot Isolation (Repeatable Read)
        
        Consistent snapshot for transaction
        
        Prevents non-repeatable reads
        
        Write skew anomalies possible
      - Serializable execution models
        
        Serializable Isolation
        
        Transactions appear executed serially
        
        Strongest isolation level
        
        Prevents all concurrency anomalies
    - - Optimistic vs pessimistic locking
        
        Optimistic Locking
        
        Assume no conflicts, check at commit
        
        Version numbers or timestamps
        
        Higher concurrency, rollback on conflict
        
        Pessimistic Locking (Two-Phase Locking - 2PL)
        
        Acquire locks before accessing data
        
        Exclusive and shared locks
        
        Lower concurrency, deadlock potential
      - MVCC implementations (Multi-Version Concurrency Control)
        
        Each write creates a new version
        
        Readers see consistent snapshot version
        
        Reduces read-write contention
        
        Used in PostgreSQL, Oracle, MySQL (InnoDB)
- - - - Forward/backward compatibility
        
        Forward compatibility
        
        Newer code reads older data
        
        Tolerate missing fields
        
        Backward compatibility
        
        Older code reads newer data
        
        Handle added fields gracefully
      - Avro schema resolution
        
        Reader and writer schemas negotiation
        
        Schema evolution rules in Avro
        
        Field addition, removal, type changes
    - - ETL pipeline architectures
        
        Extract (from source systems)
        
        Database connectors
        
        API integrations
        
        File parsing (CSV, JSON)
        
        Transform (data cleaning, mapping, aggregation)
        
        Data quality checks
        
        Data normalization and standardization
        
        Data enrichment and joining
        
        Load (into target data store)
        
        Batch loading
        
        Streaming ingestion
      - CDC implementation patterns
        
        Database log mining
        
        Reading transaction logs
        
        Identifying data changes
        
        Trigger-based CDC
        
        Database triggers for change capture
        
        Polling-based CDC
        
        Periodically querying for updates
        
        Less efficient for real-time changes
- - - - Lambda architecture evolution
        
        Separate batch and stream layers
        
        Complexity in maintaining two pipelines
        
        Data duplication and consistency challenges
      - Stateful stream processing (Kappa/Delta Architectures)
        
        Single stream processing engine for batch and stream
        
        Simplified architecture, unified processing
        
        Flink, Kafka Streams, Spark Streaming
    - - Privacy-preserving design
        
        Differential privacy
        
        Federated learning
        
        Anonymization techniques
      - Audit trail requirements
        
        Data provenance tracking
        
        Immutable data storage
        
        Compliance regulations (GDPR, CCPA)
  - - - Polyglot persistence patterns
        
        Choosing best DB for each microservice
        
        Mixing relational, NoSQL, graph DBs
        
        Data integration challenges across diverse systems
      - Derived data synchronization
        
        Message queues (Kafka, RabbitMQ) for event distribution
        
        Event streams for real-time updates
        
        Change data capture mechanisms
    - - Cryptographic auditing
        
        Merkle trees for data integrity verification
        
        Blockchain for tamper-proof logs
        
        Zero-knowledge proofs for data privacy
      - Data lineage tracking
        
        Metadata management systems
        
        Graph-based lineage visualization
        
        Data governance and compliance
- - - - Database Sharding
        
        Consistent hashing implementation
        
        Rendezvous hashing for lower overhead
        
        Maglev hashing for fast lookup
        
        Using Ketama hashing
        
        Virtual nodes for load balancing
        
        Consistent hash rings with 100s of v-nodes per physical node
        
        DynamoDB's virtual node approach
        
        Using jump consistent hash for minimal key remapping
      - Stateless Web Tier
        
        Session management strategies
        
        Client-side session storage (cookies, JWT)
        
        Shared cache (Redis, Memcached) for session data
        
        Database-backed session persistence
        
        Dynamic server provisioning
        
        Kubernetes autoscaling
        
        AWS Auto Scaling Groups
        
        Custom scripts based on CPU/memory metrics
    - - Hardware upgrade limitations
        
        Cost increase exponentially with top-tier hardware
        
        Downtime during upgrades
        
        Single point of failure remains
      - Single-node performance optimization
        
        CPU core upgrades
        
        RAM increase for larger in-memory caches
        
        Faster storage (NVMe SSDs) for disk-bound operations
  - - - Multi-Data Center Deployment
        
        GeoDNS routing
        
        Latency-based routing
        
        Geographic proximity routing
        
        Failover routing policies
        
        Async data synchronization
        
        Database replication lag considerations
        
        Eventual consistency models
        
        Conflict resolution strategies for write conflicts
      - Service Replication
        
        Master-Slave DB pattern
        
        Read replicas for query offloading
        
        Failover promotion process
        
        Split-brain scenarios and prevention
        
        Messaging queue redundancy
        
        Mirrored queues in RabbitMQ
        
        Kafka replication factor settings
        
        Active-passive broker setup for failover
    - - Automated Rollback
        
        Transactional snapshots
        
        ZFS snapshots for file system rollback
        
        Database transaction logs for point-in-time recovery
        
        Consistent snapshots across distributed storage
        
        Log-based recovery
        
        Write-ahead logging (WAL) in databases
        
        Operation logs for event sourcing systems
        
        Change data capture (CDC) for real-time updates
      - State Restoration
        
        Checkpoint mechanisms
        
        Periodic checkpoints in Apache Flink
        
        Consistent checkpoints in distributed databases
        
        Saving game state in online gaming servers
        
        Consistent hashing rebalancing
        
        Smooth rebalancing algorithms
        
        Handling in-flight requests during rebalancing
        
        Data migration strategies during node changes
- - - - Query Optimization
        
        Indexing strategies
        
        B-tree indexes for range queries
        
        Hash indexes for point lookups
        
        Composite indexes for multi-column queries
        
        Connection pooling
        
        HikariCP for Java applications
        
        c3p0 connection pool
        
        pgBouncer for PostgreSQL connection management
      - Scalability Extensions
        
        Read replicas
        
        MySQL replication
        
        PostgreSQL streaming replication
        
        Read-heavy workload distribution
        
        Connection throttling
        
        Database connection limits
        
        Queueing connection requests
        
        Circuit breaker for overloaded databases
    - - Key-Value Stores
        
        Redis caching patterns
        
        Cache-aside pattern
        
        Write-through cache
        
        Cache invalidation strategies (TTL, LRU)
        
        Leaderless replication
        
        DynamoDB replication model
        
        Cassandra's tunable consistency
        
        Conflict resolution with vector clocks
      - Document Databases
        
        Schema flexibility
        
        MongoDB's schemaless nature
        
        Couchbase JSON documents
        
        Handling evolving data structures
        
        Map-Reduce operations
        
        Hadoop MapReduce for large-scale data processing
        
        MongoDB aggregation framework
        
        Spark for distributed data transformations
  - - - Content Delivery Networks
        
        Edge server placement
        
        Geographically dispersed servers
        
        Proximity to high user density areas
        
        Peering agreements for better network paths
        
        Cache invalidation
        
        Time-to-live (TTL) based invalidation
        
        Content versioning for immediate invalidation
        
        Purging specific URLs programmatically
      - In-Memory Stores
        
        Eviction policies
        
        Least Recently Used (LRU)
        
        Least Frequently Used (LFU)
        
        First-In-First-Out (FIFO)
        
        Cache warming
        
        Pre-populating cache during startup
        
        Anticipatory caching based on access patterns
        
        Scheduled refresh of frequently accessed data
    - - Stream Processing
        
        Kafka topic partitioning
        
        Partitioning by key for data locality
        
        Topic rebalancing for even distribution
        
        Consumer group offsets for fault tolerance
        
        Exactly-once semantics
        
        Kafka transactions
        
        Idempotent processing logic
        
        Message deduplication techniques
      - Batch Processing
        
        MapReduce jobs
        
        Hadoop for offline data analysis
        
        Spark for in-memory batch processing
        
        Use cases like log analysis, data warehousing
        
        DAG workflow orchestration
        
        Apache Airflow for defining data pipelines
        
        Luigi for complex dependency management
        
        AWS Step Functions for serverless workflows
- - - - ZooKeeper Implementations
        
        Ephemeral nodes
        
        Detecting service failures
        
        Leader election in distributed systems
        
        Dynamic configuration updates
        
        Watch mechanisms
        
        Real-time updates on service changes
        
        Configuration change propagation
        
        Triggering actions based on events
      - Consistent Hashing
        
        Virtual node weighting
        
        Load balancing across heterogeneous servers
        
        Capacity proportional node assignment
        
        Handling server capacity imbalances
        
        Hotspot mitigation
        
        Randomized virtual node assignment
        
        Splay tree based consistent hashing
        
        Jumping consistent hash for minimal key movement
    - - Pub-Sub Patterns
        
        Topic partitioning
        
        Horizontal scaling of message processing
        
        Ordered message delivery within partition
        
        Parallel consumer processing
        
        Consumer groups
        
        Load balancing across consumer instances
        
        Message delivery guarantees per group
        
        Scaling consumer capacity independently
      - Queue Systems
        
        Visibility timeouts
        
        Handling message processing failures
        
        Preventing duplicate message processing
        
        Job retries after timeout
        
        Dead-letter queues
        
        Debugging failed message processing
        
        Analyzing message processing errors
        
        Retrying failed messages after investigation
  - - - Rate Limiting
        
        Token bucket algorithm
        
        Allowing burst traffic within limits
        
        Configuring refill rate and bucket size
        
        Granular control over request rates
        
        Distributed counters
        
        Redis counters for shared rate limits
        
        Consistent counter updates under concurrency
        
        Atomic increment/decrement operations
      - Circuit Breakers
        
        Failure thresholds
        
        Error percentage for opening circuit
        
        Consecutive failures for circuit trip
        
        Adaptive thresholds based on error history
        
        Half-open state
        
        Probing backend after timeout
        
        Gradual recovery process
        
        Testing backend health before full recovery
    - - WebSocket Management
        
        Connection pooling
        
        Reusing WebSocket connections
        
        Reducing connection overhead
        
        Optimizing for high concurrency
        
        Ping-pong heartbeats
        
        Detecting broken connections
        
        Maintaining keep-alive for long-lived connections
        
        Automatic connection restart
      - REST Optimization
        
        Pagination strategies
        
        Cursor-based pagination for efficient large datasets
        
        Offset-limit pagination for simpler cases
        
        Keyset pagination for consistent results under updates
        
        Field masking
        
        GraphQL query selections
        
        Field parameter in REST APIs
        
        Reducing data transfer for specific use cases
- - - - Time-Series Databases
        
        Rollup aggregation
        
        Hourly, daily, weekly aggregations for trend analysis
        
        Downsampling high-resolution data for long-term storage
        
        Reduced storage footprint for historical metrics
        
        Retention policies
        
        Storing detailed metrics for recent periods
        
        Aggregated metrics for longer historical analysis
        
        Automated purging of old data
      - Log Aggregation
        
        ELK stack patterns
        
        Elasticsearch for indexing and searching logs
        
        Logstash for log parsing and enrichment
        
        Kibana for visualization and dashboards
        
        Structured logging
        
        JSON format for logs
        
        Key-value pairs for easy parsing
        
        Standardized log fields for analysis
    - - Anomaly Detection
        
        Baseline computation
        
        Moving averages for seasonality
        
        Statistical models for normal behavior
        
        Dynamic baseline adjustments
        
        Moving percentiles
        
        P99 latency monitoring
        
        P95 resource utilization
        
        Detecting outliers beyond typical ranges
      - Escalation Protocols
        
        On-call rotations
        
        PagerDuty for incident management
        
        OpsGenie for alerting and scheduling
        
        Rotating on-call responsibilities
        
        Alert correlation
        
        Grouping related alerts to reduce noise
        
        Identifying root causes of incidents
        
        Prioritizing alerts based on impact
  - - - Blue-Green Deployments
        
        Traffic shifting
        
        Load balancer weight adjustments
        
        Gradual traffic migration to new version
        
        Zero-downtime deployments
        
        Rollback scenarios
        
        Instant rollback to previous version
        
        Automated failure detection and rollback triggers
        
        Fast recovery from deployment issues
      - Canary Releases
        
        Metrics-based validation
        
        Error rate monitoring on canary instances
        
        Performance benchmarks for new version
        
        A/B testing with canary deployments
        
        Percentage rollouts
        
        Gradual rollout to small user subsets
        
        Iterative rollout phases based on validation
        
        Controlled risk introduction
    - - Terraform Patterns
        
        Module composition
        
        Reusable infrastructure components
        
        Encapsulation of configuration logic
        
        Simplified management of complex infrastructure
        
        State locking
        
        Preventing concurrent infrastructure changes
        
        Ensuring consistent state management
        
        Collaboration safety for infrastructure teams
      - Configuration Drift
        
        Immutable infrastructure
        
        Replacing instances instead of modifying in place
        
        Reduced configuration inconsistencies
        
        Predictable environment configuration
        
        Convergence checks
        
        Automated checks for desired configuration state
        
        Drift detection and remediation
        
        Configuration management tools (Ansible, Puppet)
- - - - Blob Management
        
        Multi-part uploads
        
        Handling large file uploads in chunks
        
        Resumable uploads in case of network interruptions
        
        Parallel uploads for faster transfer
        
        Content-addressable storage
        
        Using content hash as storage address
        
        Data deduplication benefits
        
        Data integrity verification
      - Version Control
        
        Snapshot differencing
        
        Storing only changes between versions
        
        Reduced storage space for revisions
        
        Efficient retrieval of specific versions
        
        Garbage collection
        
        Automatic removal of old, unused versions
        
        Storage space reclamation
        
        Configurable retention policies
    - - Conflict Resolution
        
        Vector clocks
        
        Detecting causal dependencies between changes
        
        Merging concurrent updates
        
        Handling distributed data modifications
        
        Last-write-wins
        
        Timestamp-based conflict resolution
        
        Simpler implementation but potential data loss
        
        Suitable for eventually consistent systems
      - Delta Encoding
        
        RSYNC algorithms
        
        Efficient synchronization of large files
        
        Transferring only changed blocks
        
        Reduced bandwidth usage
        
        Binary patching
        
        Creating small patches for file updates
        
        Efficient software updates
        
        Bandwidth savings for distribution
  - - - Presence Management
        
        Heartbeat tracking
        
        User activity monitoring
        
        Session timeouts for inactivity
        
        Detecting user online/offline status
        
        Grace period handling
        
        Allowing temporary network disruptions
        
        Delaying offline status updates
        
        Handling transient connectivity issues
      - Message Sync
        
        Idempotent receivers
        
        Handling duplicate message delivery
        
        Ensuring message processing only once
        
        Unique message IDs for deduplication
        
        Causal ordering
        
        Maintaining message order based on cause-effect relationships
        
        Consistent chat history across devices
        
        Logical time ordering for messages
    - - Adaptive Bitrate
        
        Buffer modeling
        
        Predicting network bandwidth changes
        
        Dynamically adjusting video quality
        
        Minimizing buffering interruptions
        
        Quality tiers
        
        Multiple video resolutions and bitrates
        
        Switching between quality levels seamlessly
        
        Adapting to varying network conditions
      - DRM Integration
        
        License servers
        
        Content encryption key management
        
        User authorization for content access
        
        Secure content distribution
        
        Content encryption
        
        AES encryption for video streams
        
        Protecting premium content from unauthorized access
        
        Compliance with content provider requirements
- - - - Latency Number Groups
        
        Memory hierarchy costs
        
        L1 cache access time ~ 0.5 ns
        
        L2 cache access time ~ 7 ns
        
        DRAM access time ~ 100 ns
        
        Network hop penalties
        
        Intra-datacenter round trip ~ 0.5 ms
        
        Cross-datacenter round trip ~ 10 ms+
        
        Inter-continental round trip ~ 150 ms+
      - Throughput Math
        
        QPS extrapolation
        
        Requests per second calculations
        
        Scaling for peak and average loads
        
        Estimating resource requirements based on QPS
        
        Burst capacity
        
        Handling sudden traffic spikes
        
        Buffer sizing for request queues
        
        Graceful degradation under overload
    - - Cloud Cost Models
        
        Reserved vs spot pricing
        
        Long-term resource commitments vs. cost savings
        
        Flexibility vs. cost optimization
        
        Trade-offs for different workload types
        
        Cross-AZ traffic
        
        Data transfer costs between availability zones
        
        Network egress charges within cloud providers
        
        Designing for AZ isolation vs. cost efficiency
      - Optimization Levers
        
        Data compression
        
        Gzip/Snappy compression for data transfer
        
        Columnar storage formats for data warehousing
        
        Reduced storage and bandwidth costs
        
        Query batching
        
        Grouping multiple database queries into one request
        
        Reduced network round trips
        
        Improved database throughput
  - - - 4-Step Framework
        
        Requirement triage
        
        Clarifying ambiguous requirements
        
        Identifying core functionalities
        
        Prioritizing scope and depth
        
        Tradeoff analysis
        
        Consistency vs. availability in distributed systems
        
        Latency vs. throughput optimization
        
        Cost vs. performance considerations
      - Communication Tactics
        
        Assumption tracking
        
        Explicitly stating design assumptions
        
        Documenting constraints and limitations
        
        Validating assumptions with the interviewer
        
        Alternatives presentation
        
        Presenting multiple design options
        
        Discussing pros and cons of each approach
        
        Justifying design choices based on tradeoffs
    - - Red Team Exercises
        
        Cascading failure simulation
        
        Simulating service dependencies and ripple effects
        
        Identifying single points of failure
        
        Testing system resilience under stress
        
        Game day scenarios
        
        Planned failure injection exercises
        
        Testing incident response procedures
        
        Practicing fault isolation and recovery
      - Resilience Patterns
        
        Bulkheads
        
        Isolating failures to specific components
        
        Preventing cascading failures across system
        
        Resource partitioning for fault containment
        
        Failover testing
        
        Simulating node failures and testing automated failover
        
        Validating redundancy mechanisms
        
        Ensuring seamless service continuity