Please enable JavaScript.
Coggle requires JavaScript to display documents.
Apache Kafka (Topic (Each topic has multiple number of partitions. (each…
Apache Kafka
Topic
-
-
Each partition has it's own logging and their log will be stored upto a retention time (2 days for example)
Each record in each partition has an offset number which identifies that record (just) in that partition. so the offset is unique in one partition not in all topic.
-
Kafka's performance is effectively constant with respect to data size so storing data for a long time is not a problem.
Consumer
Consumer can read each part of the stream that it wants just by saving the offset of last read record
Just by using the offset consumer can skip some records and read the latest or reset the offset to some older records
-
APIs
-
-
Stream processor
Transformers: let you to consume 1 or more topics, process the records of streams and produce new stream records, then publish them to 1 or more topics
-
-
Consumers
-
-
-
If all consumers are in one consumer group then the records would be load balanced (because each record will be published to just one instance between all)
If each instance is in one consumer group (and all of them are subscribed for the topic, then the record will be published to all of the instances (because each is the agent of that consumer group)
stream platform
1- Publish and subscribe to streams of records, similar to a message queue or enterprise messaging system
-
-
-
Applications
1- Building real-time streaming data pipelines that reliably get data between systems or applications
-
