Please enable JavaScript.
Coggle requires JavaScript to display documents.
Encoding and Evolution - Coggle Diagram
Encoding and Evolution
Changes
Data format/Schema
Lead to code changes
Cannot happen at once
Server-Side apps - Rolling upgrade
Deploying to few nodes at a time
Monitor if runs smoothly
No service downtime
Encourages
More frequent updates
Better evolvability
Client-Side apps
At mercy of the client
May or may not install
Old and new v's of code and data coexist
To ensure compatibility
Backwards compatibility
Older versions read and process data written by newer versions
Relatively easy
Forwards compatibility
Newer versions read and process data written by older versions
Harder - requires older code to
ignore newer data additions
keep new field intact even though it didn't interpret it
Formats for Encoding Data
Representations
In-memory
Data kept in
Structs
Lists
Objects
Arrays
Hash tables
Trees
Optimized for eff access and manip by CPU
Typically using pointers
Sequence-of-bytes
Write to a file
Send over the network
Encode it as self-contained sequence of bytes e.g. JSON doc
Translation between the 2
Encoding
In-memory -> byte sequence
Aka serialization/marshalling
Decoding
Byte sequence -> in-memory
Aka deserialization/demarshalling
Sometimes not needed, when
DB operates on compressed data loaded from disk
Using zero-data formats
Used both on runtime and on disk/on the network
Examples
Cap’n Proto
FlatBuffers
Language-specific formats
Built-in encoding support
Bad idea to use except for minor tasks
3rd-party libraries
Convenient
Problems
Encoding is tied for a specific lang
Decoding in another lang is difficult
Insecure decoding may expose vulnerabilities
Often neglect versioning data > b/f compatibility probs
Not CPU efficient
Standardized formats
Human-readable
CSV
Only supports tabular data w/o nesting
Ambiguity around numbers
No schema => app defines meaning of rows/cols
Parsers inconsistently interpret commas and newlines
XML
Too verbose
Unnecessarily complicated
Ambiguity around numbers
JSON
No diff b/w int and float
No precision
Problem with large nums (>2^53)
Schemas include standard primitive types
Allows devs to explic overlay constraints on fields
Schema content model
Open
Any undefined field to have any datatype
By default (having additionalProperties = true)
Schema define what is invalid and not what is valid (constraints)
More powerful
More complex
Closed
Only allows fields explicitly defined
Also supports
if/else schema logic / types
Refs to remote schemas
Less verbose than XML
JSON and CSV
No support for binary strings -> encode data as text using Base64 => 1/3 more space
Good support for Unicode str
Powerful => complicated
App may need to hardcode enc/dec logic
Widely used and supported
Binary encodings
Plethora of such for JSON and XML
More compact & faster to parse
Niche
Some of them extend datatypes
JSON binary encodings 81 - 66 bytes
Include the encoded field names
Schema-based
Protocol buffers (protobuf)
Has code generation tool
Implements schema as classes in various languages
No support for add. restriction like the JSON schema
33 bytes
No field names
Field tags
Reference schema definition
Are critical
Can change schema field name
but can't change a field's tag => all existing encoded data invalid
Schema evolution
Can add schema field w/ a new tag number (never been used bf)
Old code ignores new unrecognized field tags => fw compatibility
New code reads old data and fills new fields with default values => bw compat
Can change dadatypes but can lead to trunc
Almost same goes true for Thrift
Requires schema for any encoded data
Avro
Schema
Uses a schema to specify the struc of enc data
2 schema languages
Avro IDL (human editing)
One based on JSON (machine-readible)
No complex constrictions on fields
No tag numbers => more flexible and allows for dyn. gen. schemas
32 bytes
No identifier field
Field order consist. w/ schema specs
=> helps decode correct types
Writer's
Used to encode data
Compiled into the app
Reader's
Used to decode data
If a field is missing from R's schema fill fith default values
If differs from Writer's Avro compares the two and translates data from W to R schema
If add. field exists in W's schema ignore it
You may add or remove only fields that have default values
Allows null by use of unions and is more constricting to avoid mistakes
Changing datatype and name of field is possible with a few caveats
Include it once in the beginning of a file when lots of records are encoded in that file
If files are indiv. written, record may begin with schema ver. number
Pros
Much more compact than BSON and the likes
Schema is a valuable documentation
Keeping a DB of schemas encourages bw and fw compatibility
Statically typed langs can make use of compile time type checks
Evolvability
Adapt to change
Modes of dataflow
Dataflow Through DB
1 process both Es & Ds
Need to ensure bw compatibility
Diff apps/services/instances
Also needs fw compatibility
Data migration
Exp. for large DS
=> Data outlives code (years old data)
Data rewritten during LSM compaction
Archival storage
Periodic snapshots
For backup
Data dump will be encoded in latest schema
Even if original was in old schema
Dataflow Through Services: REST and RPC
Clients
Connect to servers to make requests to API
Make GET req to download content
Make POST req to submit data to server
Servers
Expose APIs
Returns response
Html for displaying to human
Data in machine-readable format (e.g. JSON)
API
Set of protocols and data formats
HTTP
When main protocol the service is called web service
REST
Builds upon HTTP
Service design philosophy
Emphasizes simple data formats
URLs for identifying resources
Using HTTP features for
Authentication
Content type negotiation
Cache control
RESTful API
Treats state transfers over network as distinct from a func call
URLs
SSL/TLS
HTML
No arbitrary queries
Only allows predefined inputs and outputs
Diff microservices API should remain compatible
IDL for documentation
Swagger
Protocol Buffers
Devs use service frameworks
Examples
gRPC
Spring
Boot
FastAPI
Takes care of
Routing
Metrics
Caching
Authentication
The RPC model
Remote procedure calls
Make remote service request look like a func call
Fundamentally flawed
Local funcs are predictable, parameters - in your control
Network requests - unpredictable, reasons - outside your control
Network request introduces return without result cause of timeout
Without idempotence you may retry a request if first one got through but only the response was lost
Latency is quite variable
Client and service may be implemented in diff languages - translation needed
Client gets ahold of service IP to connect to
Load balancers
Hardware
Pieces of equipment installed in datacenters
Clients connect to a single host and port
Requests are routed
Software
Same but an application, not equipment
DNS
Supports load balancing
By allowing multiple IP add's to be asoc. with same domain name
Client configured to connect to domain name, not ip address
Caches for long periods of time so new changes are delayed
Service discovery
Supports dynamic addition/removal/changes to services
Gives clients more metadata
Service meshes
Software LB + Service Discovery
Typically deployed as a process on both client and server
Complicated
Encryption handled at LB level
Sophisticated observability
Durable Execution and Workflows
Service-based architectures have multiple services all responsible for diff parts of app
The sequence of steps = workflow
Each step is a task/activity/durable function
Workflows
Defined as graph of tasks
Definition may be written in
A general purpose programming lang
DSL (Domain Specific Language)
Markup lang like BPEL
Executed by workflow engine
When and on which machine to run each task
How to handle task failures
How many tasks can run parallelly
Composed of
Orchestrator
Schedules tasks
Triggers the workflow if user specified times
Executor
Executes tasks
Use cases
Integrate with data systems and orchestrate ETL tasks
Provide graphical notation for workflows
Provide durable execution
For architectures that require transactionality
Provide exactly-once semantics for workflows
Log all RPCs and state changes
RPC order must remain same
Sensitive to code changes
Execute only tasks that haven't been executed
Execution begins when a workflow is trigged
Event-Driven Architectures
Request is called event/mesasge
A way encoded data to flow from one process to another
Unlike RPC sender doesn't wait for response from recipient
Events not sent through network via direct connection, but via an event broker which stores event temporarily
>
direct RPC
Decouples sender from recipient
Same message - several recipients
Avoid need for service discovery
Automatically redeliver messages to crashed process
Buffer improving reliability
Asynchronous, but possible to make it sync
Message distribution patterns
Queue/Consumer
One consumer receives message
Topic/subscriber
Every subscriber receives the message
Message brokers
Don't enforce data model
Deploy a schema registry alongside it
Differ in durability
Many write to disk so if crash it persists
Many automatically delete messages after being delivered
Can be config to store indefinitely
Distributed actor frameworks
Actor model
Actors instead of threads
Each actor has its own private state
Actors talk by sending async messages
Actor behavior
One message at a time
No locks, no shared state
Some messages can be lost
Distributed actors
Same message passing local or remote
Framework encodes, sends, decodes messages over network
Location of actors is hidden (location transparency)
Pros
Concurrency + distribution in one model
Built‑in message broker behavior
Independent scheduling of actors
Versioning / upgrades
Old and new nodes talk via messages
Messages must be forward/backward compatible
Use robust encodings (schemas) for messages