Please enable JavaScript.
Coggle requires JavaScript to display documents.
The Trouble with Distributed Systems - Coggle Diagram
The Trouble with Distributed Systems
Unreliable networks
Shared-nothing systems, network-only communication
Asynchronous packet networks, no delivery guarantee
Indisting. to sender
Request lost in transit
Request queued, delayed delivery
Remote node has failed
Remote node temporarily unresponsive
Response lost on return
Response processed but delayed
TCP's Limits
Breaks streams into packets, reassembles
Detects loss, reorders, checksums data
Congestion/flow control manages send rate
Can't tell which packet was lost
Ack ≠ app processed the request
Faults in Practice
~12 network faults/month per datacenter
Redundancy doesn't fix human error
Cows, beavers, sharks cut fiber
Cross-region RTTs can hit minutes
Asymmetric partitions (A-B, B-C, not A-C)
Brief outages cause lasting repercussions
Fault Detection
Closed ports send RST/FIN signals
Crash scripts can notify peers fast
Switch/router feedback often unavailable
ICMP unreachable not fully reliable
Timeouts
Only real fallback method
Long timeout = slow failure detection
Short timeout = false dead nodes
Premature death → duplicate actions
Bounded delay: 2d + r (theoretical)
Overload → cascading failure risk
Real networks: unbounded, no guarantee
Congestion & Queueing
Switch queues fill under contention
CPU/thread busy delays processing
VM pausing buffers incoming data
TCP send-side queueing too
TCP vs UDP: reliability vs latency
Delay Variability
Multitenant clouds add noisy neighbors
Timeouts set experimentally, not fixed
Phi Accrual: adaptive timeout detection
Sync vs Async Networks
Telephone circuits
Fixed bandwidth
Bounded delay
Packet switching
Dynamic
Bursty-traffic optimized
Circuits waste capacity when idle
QoS/ATM/InfiniBand: hybrid attempts
Trade-off: utilization vs predictability
Faults and Partial Failures
Single PC behavior
Deterministic hardware and software
Either fully functional or broken
Prefer crash over wrong results
Hardware faults → total failure
Wrong results rare, usually ignored
Distributed Systems behavior
Faults frequent, can’t ignore them
System can be partly broken
Operations sometimes succeed, sometimes fail
Outcomes may be unknowable to caller
Nondeterministic behavior
Engineering mindset
Consider wide range of faults
Even super rare ones
Simulate failures in tests
Suspicion and pessimism help
Design for graceful degradation
Assume partial failure as normal