Please enable JavaScript.
Coggle requires JavaScript to display documents.
11 Event Driven Systems Book Part 1 - Coggle Diagram
11 Event Driven Systems
Book Part 1
Finance industry regulation (Jan 2018)
Trading activity has to be reported to a regulator within one minute of it happening.
Event Sourcing
Replayable logs
Central point of storage that is fault-tolerant and scalable—
a shared source of truth that any application can fall back to.
Mindset change from Synchronous to Asynchronous
Scaling database-centric systems.
Event Sourcing
CQRS
ToC
Part I => Streaming and how Kafka works.
Part II => Patterns: Event Sourcing, Event Collaboration, CQRS
Part III => Microservices and SOA, with a focus on event streams as a source of truth
Part IV => building a small streaming system using Kafka Streams (and KSQL).
Core mantra of event-driven services
Centralize an immutable stream of facts.
Decentralize the freedom to act, adapt, and change.
Principles
Decoupling
Plug / unplug systems into the real time stream of events.
Kafka
Streaming platform
Kafka Cluster
Data transfer and storage,
provided at scale and high availability.
A Kafka cluster is a distributed system at heart, providing high availability, storage, and linear scaleout.
One Big Cluster (200-nodes) 2. Multiple Clusters
Multitenant
Protect against denial-of-service attacks,
causing service degradation or instability.
Quotas (Bandwidth amount)
Strong Ordering Guarantees
Kafka provides ordering guarantees
only within a partition.
Supply an unique id (CustomerId as their partitioning key)
For global ordering
use a single partition topic.
Retry
Send (message) batches one at a time, per destination machine, so there is no potential for a reordering of events when failures occur and batches are retried.
Database
KSQL, Kafka’s SQL-like
stream processing language.
Compute queries continuously, and emitted if the value of the query has changed
Storage;
production topics with hundreds of terabytes
Data held in the log.
These can be piped into views that users can query directly.
supports transactions.
It’s about real-time processing first,
long-term storage second.
Originally built to distribute the datasets
less conventional use cases
data storage, 2. dynamic failover, and 3. bandwidth protection
Producer (/Partitioner)
The partitioner will either spread data across the available partitions in a round-robin fashion or, if a key is provided with the message, use a hash of the key to determine the partition number. This latter point ensures that messages with the same key are always sent to the same partition and hence are
strongly ordered.
Durability
Through replication (3 copies)
Best practice: configure the producer to wait
for replication to complete
Compacted Topics
Data that has a primary key (identifier)
Good for stateful stream processing.
These compacted topics retain only the most recent events,
with any old events, for a certain key, being removed.
Asynchronous process
Long-term data storage
Retention-based or compacted topics
holding more than 100 TB of data.
Security
1.TLS 2.Permission 3.Quota