Please enable JavaScript.

Coggle requires JavaScript to display documents.

Optimizing Kafka Deployment - Coggle Diagram

- - - - + No. of Topic Partitions
        
        Partitions are the unit of parallelism in Kafka
        
        Take care of
        
        producer and consumer throughput
        
        key assignments - messages with the same key go to the same partition
        
        Latency
    - - Batching Strategy
        
        + batch.size
        
        the max size in bytes for each message batch
        
        + linger.ms
        
        Increase the possibility of batching - the waiting time for the producer to wait for batch to reach the configured batch.size
        
        Take care of
        
        Latency
        
        as the producer is waiting for messages to be batched, messages are not sent immediately
      - Compression
        
        compression.type
        
        lz4, snappy, zstd, and gzip
        
        lz4 is recommended
        
        Take care of
        
        Batching
        
        more batching results in better compression
        
        compression codecs are matching for the producer and topic
        
        Latency
      - acks = 1
        
        waiting for the leader broker to send acks responses may impact the producer throughput
        
        acks = 1 will guarantee that the partition leader has received the message
        
        Take care of
        
        Durability
        
        producer will not wait until the message is replicated
      - buffer.memory
        
        Kafka producers automatically allocate memory for the Java client to store unsent messages.
        
        If that memory limit is reached, then the producer will block on additional sends until memory frees up or until max.block.ms time passes.
        
        only needed in case of Large number of partitions
    - - + fetch.min.bytes
        
        corresponding to batch.size
        
        the minimum number of bytes expected for a fetch response from a consumer
        
        Take care of
        
        Latency
      - Consumer Groups
        
        Coordination of the consumers
        
        Take care of
        
        No. of Partitions
        
        no of consumers in consumer group = no. of partitions
- - - - - No. of Partitions
        
        it may take longer to replicate a lot of partitions shared between each pair of brokers and consequently take longer for messages to be considered committed.
      - num.replica.fetchers
        
        Increase if followers can't keep up with the leader
    - - Batching Strategy
        
        - batch.size
        
        - linger.ms
      - Disable Compression
        
        disabled by default
      - acks = 1 or 0
        
        Take care of
        
        Durability
        
        producer will not wait until the message is replicated
    - - - fetch.min.bytes
    - - Local Stream Processing
        
        instead of joins with external databases
        
        use Kafka Connect to get the data needed from the external database and import it to Kafka
- - - - Replication
        
        replication.factor
        
        the no. of replicas of the partition
        
        Take care of
        
        No. of Brokers
        
        default.replication.factor
        
        if auto.create.topics.enable = true
      - min.in.sync.replicas
        
        with acks = all
        
        Take care of
        
        Availability
      - broker.rack
        
        spreads replicas of the same partition across different racks
      - unclean.leader.election.enable=false
        
        to ensure that new leaders should be elected from just the ISR list.
        
        This prevents the chance of losing messages that were committed but not replicated.
        
        Take care of
        
        Availability
    - - acks = all
        
        ensure that the message is replicated to all ISR
        
        Take care of
        
        Throughput
        
        Latency
      - retries
        
        delivery.timeout.ms
        
        Take care of
        
        Order
        
        max.in.flight.requests.per.commection
        
        retries=0 && Callback
        
        Duplication
        
        enable.idempotence=true
        
        to prevent duplication
    - - enable.auto.commit=false
        
        Only commit after the message is completely processed using commitSync()
      - isolation.level=read_commited
        
        when using EOS
- - - - No. of Partitions
        
        more partitions increase recovery time due to leader election time for each partition
      - min.in.sync.replica
        
        . In the case of a shrinking ISR, the higher this minimum value is, the more likely there is to be a failure on producer send, which decreases availability for the partition.
      - unclean.leader.election
      - num.recovery.threads.per.data.dir
        
        The number of threads per data directory to be used
        for log recovery at startup and log flushing at shutdown
    - - - session.timeout.ms
        
        The lower the session timeout is set,
        the faster a failed consumer will be detected
      - + max.poll.interval.ms
        
        to avoid soft failures
    - - num.standby.replicas
        
        If a local state store does not exist, i.e., num.standby.replicas=0, the changelog is replayed from the earliest offset.
        
        If a local state store does exist,
        the changelog is replayed from the previously checkpointed offset.