Please enable JavaScript.
Coggle requires JavaScript to display documents.
Understand Kafka Topic Partitions - Coggle Diagram
Understand Kafka Topic Partitions
messages are not pushed to consumers, instead, consumers have to pull messages
Concepts
Broker
Events
A fact that happened
Streams
represent related events in motion
Topics
Group related events together and durably stores them
Consumer pulls messages off of a Kafka topic while producers push messages into a Kafka topic
May have many producers and consumers
Like DB tables or Dir in file systems
Partitions
A topic is divided into several partitions
Is the smallest storage unit that holds a subset of records owned by a topic
Single log file where records are written to it in an append-only fashion
Are the way that Kafka provides scalability
Distributes the partitions of a particular topic across multiple brokers
Although the messages within a partition are ordered, messages across a topic are not guaranteed to be ordered
The records in the partition are each assigned a sequential identifier called
offset
Redundancy
: Kafka keeps more than one copy of the same partition across multiple brokers
Write records to partitions
Using partition key
Allowing Kafka to decide the partition
Writing a custom partitioner
Be sure that the records are well distributed across partitions
Reading records
Consumer groups ensure that a message is only ever read by a single consumer in the group
Consumer connects to a partition and reads message
The offset is uesed to keep track
can resume after crash - consumer responsibility to keep it
Everything in Kafka revolves around partitions
