Please enable JavaScript.

Coggle requires JavaScript to display documents.

Apache Kafka A high-throughput distributed messaging system (內容 (基本概念…

- - - - Queue
      - Pub/Sub
    - - Scability
        
        水平擴展
        
        Topic is partitioned(shared)
      - High Availability
        
        Partition 可以被 replicated
      - High Throughput
      - Durability
        
        所有資料都被保存(persisted)
        
        循序讀寫(類似log file)
        
        消費端會保存消息的編移(offset) 類似 file descriptor
        
        log 檔案可以被 rotated(類似logrotate)
        
        過期的消息只能被刪除(類似logrotate)
        
        支援批次載入和即時使用
  - - - Producer: 發送消息到 Broker
      - Consumer: 從 Broker 消費消息
      - Broker
        
        Kafka Cluster 中的一個節點
      - ZooKeeper: 協調 Kafka Cluster 和 Consumer Group
    - - Topic
        
        The named destination of partition?
      - Partition
        
        一個 topic 可以有多個 partition
        
        平行處理單位
      - Message
        
        Key/Value Pair
        
        Message Offset
      - 情境
        
        Queue(One Partition One Consumer)
        
        Pub/Sub(One Partition Multiple Consumer)
        
        Multiple Partition
    - - Group of Worker
      - 共享 offset
      - offset 同步到 zooKeeper
      - 自動平衡負載
    - - Replication單位是 Partition
      - 每個Partition有一個Leader和0+ Follower
      - 讀寫都是透過 Leader
    - - Broker to Consumer
        
        At least once - Store the offset after handling the message
        
        Exactly once - At least once + Idempotent operation
        
        At most once - Store the offset before handling the message
      - Producer to Broker
        
        At most once - Async send
        
        At least once - Sync send (with retry count)
        
        Exactly once! - Idempotent delivery (v0.9+)
      - 情境
        
        At most once: Messages may be lost but are never redelivered.
        
        At least once: Messages are never lost but may be redelivered
        
        Exactly once: each message is delivered once and only once. (this is what people actually want)
  - - - Stream Processing Framework
        
        Strom
        
        Samza
        
        Spark Streaming
    - - Logstash
      - Flume