Please enable JavaScript.
Coggle requires JavaScript to display documents.
Architecting Distributed Cloud Applications (networking (idempotency…
Architecting Distributed Cloud Applications
Orchestrators
Manage cluster's features
lifecicle
networking
upgrades
scaling up and down
deploys and runs services
Containers
like a virtual machine but more ligthweight
minimal peace of run code
container image
immutable
defines version of code
with dependencies
use the same image in all environments
development, test, staging, production
runs an image in isolated environment
more isolated then single process
but less isolated then virtual machine
share:
hardware
OS Kernel
does not share:
resources (file system)
Hyper-V container
does not share OS Kernel
easier in terms of DevOps
CI/CD
container can be fast
deployed to environment
Region - availability zone
fault domain - unit of failure
Hieracy
Planet/Region/Availability zone/Rack/PC/VM
Private fiber-optic network among zones
Microservices
Benefits
Scale independently
Develop independently
Different technology stack
Many clients can use one service
Each service can use appropriate version of library
SLA
SLA(System) = Multiply(SLA(service))
Each service SLA = 99.9
SLA of system = 99.9
99.9
... * 99.9
Total downshift time = Sum(downshift time of each service)
Autoscalling
By queue length
By consumed resources
By schedule
Holudays/weekends/etc
12 factors services
Single root repo. Don't share code with another services
Deploy dependent lib with services
No config in code - external configuration
Strictly separate build and run stages
Treat backing services as attached resources
Execute the app as one or more stateless processes
Embracing failure
a lot of reasons for failure
application should be ready for it
networking
networking communication
should be explicit
don't use RPC
is not reliable
latency is not zerro
is not secure
topology is dynamic
(ip address can be changed)
endpolint
it is instance of a service (ip:port)
virtual many services in one port
because of VM/containers
to scale-up and high availability
used reverse proxy
with many endpoints behind it
forward proxy
content filtering
caching
logging
API versioning
new API must always be backward compatible
use API version in URL (site.ru/v1.1/product/2)
data contract should be language agnostic
simple fields
openapis.org
swagger.io
idempotency
operation can be executed many times
without any side effects
client can retry operation
because
network fallacies (timeout, ip changed)
server need to throttle invocations
querying is idempotent naturally
for methods with side effects use dedublications
:question: what the best way to get idempotency?
messaging
Resource efficient
Client does not wait
service pulls work instead push work to service
service does not listen
Resilient
if service fails another
instance can do his work
service can be offline and
will do work in the future
Elastic
use queue length to determine
need to scale up/down
simple add new instances
to do work
message can be processed 1+ times and out of orrder
dequeue message
DequeueCount++
hide message for n secconds
if DequeueCount > threshold - log bad message and delete it
else OK and delete message
can be send to multiple subscribers
has time to leave and after this time - deleted
service can update invisibility timeout
using this feature we can process message only once :question:
Service upgrade and config
Delete and update
Stop cluster, delete it and startup new one
service became unavailable while update
Rolling update
stop some instance of service inside cluster
startup new instances of service inside cluster
2 phase update API - contracts
Blue-Green deployment
Reverse proxy orchestrates new and old instances
of service inside cluster
some orchestrates can shutdown service with gracefully
and give time to completes all requests
Don't use update configurations on-fly
it is complex and inconsistent
Leader election
via database
each service update a flag in shared table
with expiration date
sometime necessary to work
just one instance of service
for example
night reports
via queue message
just one service can handle the message
Data storage
File storage service
documents, audio, video..
usually inexpensive
offer public read-only access
it allows read data directly from storage
without services
to improve performance can be used
Content Delivery Network (
CDN
)
Relational (SQL) databases
Easy query, join, indexing, sorting etc
Hard to scale
Non-relational (NoSQL) databases
Easy to scale
Hard to get joined related information
Data partitioning
We need to know criteria of partitioning
it is hard because in the future this criteria
can change
Replication
Every peace of data should be saved in many databases
Consistency
Two records should store both or not store any
ACID transactions:
Atomicity
Consistency
Isolation
Durability
CAP theorem
In distributed systems you can achieve
Consistency or Availability
but not both
CQRS
(Command Query Responsibility Segregation)