Please enable JavaScript.

Coggle requires JavaScript to display documents.

System Design (Distributed transactions (Saga (How to reduce complexity…

- - - - 1. Compensatable transactions
      - 2. Pivot transaction - most important one, that cannot be compensated
      - 3. Retriable transactions
    - - Reorder steps to minimize business risk
      - Bounded context
      - Hide intermediate (not completed transaction) data with flags or statuses
      - Use CQRS or data replication for local consistency constraints
      - Avoid synchronous communication between services
      - Rely on retriable idempotent operations
      - Prefer optimistic locking with data versioning
      - Use distributed locking tools for critical cases
    - - Choreography
      - Orchestration
  - - - CDC tools - automate database changes capture
        
        debezium - transaction log miner
- - - - Definition
        
        data is stored durably, in order, and can be read deterministically
        
        publish/subscribe messaging system
      - Consists of
        
        Message
        
        is simply an array of bytes, so the data contained within it does not have a specific format or meaning to Kafka
        
        Key
        
        Example: generate hash of the key, and then select the partition by it. This assures that messages with the same key are always written to the same partition.
        
        is also a byte array and has no specific meaning to Kafka
        
        Optional bit of metadata of message
        
        Batch
        
        For efficiency, messages are written into Kafka in batches
        
        collection of messages, all of which are being produced to the same topic and partition
        
        typically compressed, providing more efficient data transfer and storage
        
        Schema
        
        messages are opaque byte arrays to Kafka itself, it is recommended that additional structure, or schema, be imposed on the message content so that it can be easily understood
        
        For example: JSON, XML, Avro
        
        A unit of data within kafka
        
        Topics and partitions
        
        Messages are categorized into topics
        
        a topic typically has multiple partitions, there is no guarantee of message time-ordering across the entire topic, just within a single partition
        
        Topics are additionally broken to number of partitions
        
        Messages are written to it in an append-only fashion, and are read in order from beginning to end
        
        Each partition can be hosted on a different server, which means that a single topic can be scaled horizontally across multiple servers
        
        partitions are the way a topic is scaled within a Kafka cluster
        
        How to choose number of partitions
        
        Number of partitions can only be increased, not decreased
        
        important to use partition counts that will balance the message load across the entire cluster as brokers are added. Many users will have the partition count for a topic be equal to, or a multiple of, the number of brokers in the cluster. This allows the partitions to be evenly distributed to the brokers, which will evenly distribute the message load
        
        What is the throughput you expect to achieve for the topic? For example, do you expect to write 100 KB per second or 1 GB per second?
        What is the maximum throughput you expect to achieve when consuming from a single partition? A partition will always be consumed completely by a single consumer (as even when not using consumer groups, the consumer must read all messages in the partition). If you know that your slower consumer writes the data to a database and this database never handles more than 50 MB per second from each thread writing to it, then you know you are limited to 50 MB/sec throughput when consuming from a partition.
        
        Producers and consumers
        
        Producers
        
        In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. This assures that all messages produced with a given key will get written to the same partition.
        
        Create new messages to a specific topic
        
        By default, the producer does not care what partition a specific message is written to and will balance messages over all partitions of a topic evenly.
        
        Consumers
        
        Subscribes to one or more topics and reads the messages in the order in which they were produced
        
        Offset
        
        keeps track of which messages it has already consumed by keeping track of the offset of messages
        
        Each message in a given partition has a unique offset.
        
        By storing the offset of the last consumed message for each partition, either in Zookeeper or in Kafka itself, a consumer can stop and restart without losing its place.
        
        The offset is another bit of metadata—an integer value that continually increases—that Kafka adds to each message as it is produced.
        
        Consumer groups
        
        Same consumer group
        
        The group assures that each partition is only consumed by one member.
        
        Consumer is exclusively mapped to one or more partitions
        
        1 more item...
        
        In this way, consumers can horizontally scale to consume topics with a large number of messages
        
        if a single consumer fails, the remaining members of the group will rebalance the partitions being consumed to take over for the missing member.
        
        Rebalance
        
        3 more items...
        
        Consumers work as part of a consumer group, which is one or more consumers that work together to consume a topic
        
        same messages won’t be received by the other consumers (assigned to different partitions)
        
        Different consumer groups
        
        providing the “publish/subscribe” pattern
        
        messages from topic partitions are sent to all the consumers across the different groups
        
        Brokers and clusters
        
        Broker
        
        A single Kafka server is called a broker
        
        receives messages from producers, assigns offsets to them, and commits the messages to storage on disk
        
        services consumers, responding to fetch requests for partitions and responding with the messages that have been committed to disk.
        
        Depending on the specific hardware and its performance characteristics, a single broker can easily handle thousands of partitions and millions of messages per second
        
        Cluster
        
        Kafka brokers are designed to operate as part of a cluster.
        
        Controller
        
        Within a cluster of brokers, one broker will also function as the cluster controller (elected automatically from the live members of the cluster)
        
        responsible for administrative operations
        
        assigning partitions to brokers
        
        monitoring for broker failures
        
        A partition is owned by a single broker in the cluster, and that broker is called the leader of the partition
        
        A partition may be assigned to multiple brokers, which will result in the partition being replicated
        
        This provides redundancy of messages in the partition, such that another broker can take over leadership if there is a broker failure
        
        However, all consumers and producers operating on that partition must connect to the leader
        
        Multiple clusters
        
        Reasons to have
        
        Segregation of types of data
        
        Isolation for security requirements
        
        Multiple datacenters (disaster recovery)
        
        ability to scale the load across multiple servers
      - Why Kafka
        
        Disk-Based Retention
        
        These options can be selected on a per-topic basis, allowing for different streams of messages to have different amounts of retention depending on the consumer needs
        
        Topics can also be configured as log compacted, which means that Kafka will retain only the last message produced with a specific key. This can be useful for changelog-type data, where only the last update is interesting.
        
        Kafka brokers are configured with a default retention setting for topics, either retaining messages for some period of time (e.g., 7 days) or until the topic reaches a certain size in bytes (e.g., 1 GB)
        
        Once these limits are reached, messages are expired and deleted so that the retention configuration is a minimum amount of data available at any time
        
        durable message retention means that consumers do not always need to work in real time
        
        Messages are committed to disk, and will be stored with configurable retention rules
        
        Durable retention means that if a consumer falls behind, either due to slow processing or a burst in traffic, there is no danger of losing data.
        
        It also means that maintenance can be performed on consumers, taking applications offline for a short period of time, with no concern about messages backing up on the producer or getting lost.
        
        Consumers can be stopped, and the messages will be retained in Kafka. This allows them to restart and pick up processing messages where they left off with no data loss.
        
        Multiple producers
        
        Kafka is able to seamlessly handle multiple producers, whether those clients are using many topics or the same topic
        
        This makes the system ideal for aggregating data from many frontend systems and making it consistent.
        
        For example, a site that serves content to users via a number of microservices can have a single topic for page views that all services can write to using a common format
        
        Consumer applications can then receive a single stream of page views for all applications on the site without having to coordinate consuming from multiple topics, one for each application.
        
        Multiple Consumers
        
        Kafka is designed for multiple consumers to read any single stream of messages without interfering with each other
        
        This is in contrast to many queuing systems where once a message is consumed by one client, it is not available to any other
        
        Multiple Kafka consumers can choose to operate as part of a group and share a stream, assuring that the entire group processes a given message only once
        
        Scalable
        
        Users can start with a single broker as a proof of concept, expand to a small development cluster of three brokers, and move into production with a larger cluster of tens or even hundreds of brokers that grows over time as the data scales up. Expansions can be performed while the cluster is online, with no impact on the availability of the system as a whole.
        
        High Performance
      - When it's used
        
        Use cases
        
        Messaging to users (email...). All micros send messages via kafka to message handling system responsible for notifications sending.
        
        Collecting metrics and logging
        
        Commit log - database changes published to kafka
        
        Stream processing, operating on data as soon as its produced
        
        Activity tracking
        
        Scenarios
        
        As scalable event streaming platform for data integration
        
        For decoupling between different producers and consumers
        
        To handle backpressure
        
        To continuously process and correlate incoming events
        
        For enabling the creation and updating of materialized views within other databases
        
        To allow interactive queries directly to Kafka
  - - - Publish/subscribe messaging is a pattern that is characterized by the sender (publisher) of a piece of data (message) not specifically directing it to a receiver. Instead, the publisher classifies the message somehow, and that receiver (subscriber) subscribes to receive certain classes of messages. Pub/sub systems often have a broker, a central point where messages are published, to facilitate this.
- - - - store data in pairs of keys and values
        
        keys have no relation or structure (so, keys could be height, age, hair color, for example)
      - When to use
        
        useful for storing fairly basic information, like details about a customer
      - products
        
        Redis
        
        Memcached
        
        Ehcache
        
        Amazon DynamoDB
    - - Separate data into discrete columns. Instead of using rows – whereby the row ID is the main key – column-based database systems make the data the main key
      - By using columns you can gain much greater speed when querying data
      - When to use
        
        most appropriate for big data and instances where data is relatively simple and consistent
      - Products
        
        Cassandra
        
        HBase
    - - Similar to key/value pair database management systems. The only difference is that the value that is paired with a key is stored as a document. Each document is self-contained, which means no schema is required
      - When to use
        
        When you need flexibility of document-oriented systems, but they are not built for complex queries
      - Products
        
        MongoDB
        
        CouchDB
        
        CouchBase
        
        DynamoDB
    - - contain the relationships between different data
        
        store data as nodes, with the ‘edges’ of the nodes describing their relationship to other nodes
      - When to use
        
        can be used to build and model networks
      - products
        
        Neo4j
- - - - Control the producer (slow down/speed up is decided by consumer)
      - Buffer (accumulate incoming data spikes temporarily) - in memory or in external message queue like kafka
      - Drop (sample a percentage of the incoming data)
      - Scaling