Please enable JavaScript.
Coggle requires JavaScript to display documents.
Cross-cluster data mirroring - Coggle Diagram
Cross-cluster data mirroring
Use-cases of cluster mirroring
regional and central clusters
Redundancy
cloud migrations
Multi-cluster Architecture
Realities of cross
data center communications
High Latencies
Limited bandwidth
High costs
Hub and spokes Architecture
Case multiple local Kafka Clusters and one local Kafka Cluster
Active-Active Architecture
Datacenters shares data and all can produce and consume
Benifits
Redundancy
serve users from nearby data centers
drawbacks
conflicts between the datacenters
when a user uses a datacenter for writing and another for reading
duplicating messages infinitely
Active-standby Architecture
supports failure case and disaster scenario
Benefits
simplicity " you don't need to worry about conflicts"
Disadvantage
waste of good cluster as it works only during failure
Failover Case
Data loss and inconsistency
start offset after failover
auto offset reset
replicate offset topic
Time based failover
external offset mapping
After the failover
scrape the original cluster and start again
Stretch Cluster
in case an entire datacenter fails
only one cluster not multi-cluster
Advantages
synchronous replication
all brokers are in use no waste
expensive hardware
three datacenters with low latency and high bandwidth
uneven number of nodes for majority
Mirror Maker
replicate data between datacenters
Other Mirroring solutions
Uber uReplicator
Confluent Replicator
Important to monitor in production
Lag monitoring
difference in offsets
two ways
check the latest committed by MM to the source
check the
metrics monitoring
Canary
every minute produces an event and consumes it
Tuning Mirror Maker
depends on throughput and lag tolerated
