Please enable JavaScript.
Coggle requires JavaScript to display documents.
Integration and Messaging - Coggle Diagram
Integration and Messaging
SQS
Standard Queue
Produce Messages
:
Use
SendMessage API
from SDK
Message persist in SQS until consumer deletes it
Examples: send an order to be processed (Order ID, Customer ID, etc)
Consume Messages
:
Consumers can be EC2 instances, AWS Lambda etc
Poll SQS for messages (receive up to
10 messages
at a time)
Delete message from SQS using
DeleteMessage API
SQS with ASG
Usage 1
:
Trigger the CloudWatch Alarm whenever CloudWatch Metric - Queue Length (ApproximateNumberOfMessages) exceeds certain level (image)
Alarm to increase the capacity of ASG by X amount
Usage 2
:
Trigger the CloudWatch Alarm whenever CloudWatch Custom Metric -
Queue Length / Number of instances
exceeds certain level (image)
Alarm to increase the capacity of ASG by X amount
Security
Encryption
Features
:
In-flight encryption using
HTTPS API
At-rest encryption using
AWS KMS keys
Client-side encryption if client wants to perform encryption/ decryption itself
Access Controls
IAM policies
to regulate access to SQS API
SQS Queue Access Policies
Features
:
Useful for
cross-account access
to SQS queues
Useful for allowing
other services
(e.g. SNS) to write to SQS queue
User cases
:
Let EC2 instance from 1 account to poll message from another Account with the SQS queue (image)
Publish S3 Event Notifications to SQS Queue (image)
Features
:
Managed service
Use to
decouple applications
(image)
Unlimited throughput, unlimited number of message in queue
Default retention of messages 4 days; maximum of 14 days
Low latency (
<10ms
on publish and receive)
Limitation of
256KB
per message sent
Can have duplicate message (at least once delivery)
Can have out of order messages (best effort ordering)
Use separate queues to provide prioritization of work
Message Visibility Timeout (image)
Features
:
Message becomes invisible to other consumers after a message is polled by a consumer
Message Visibility Timeout is
30 seconds
by default
Meaning that message has 30 seconds to be processed
Message is Visible again in SQS after the Message Visibility Timeout is elapsed and message has not been deleted
Consume can call ChangeMessageVisibility API to extend the Message Visibility Timeout
Cons
:
Re-processing take longer
, if Message Visibility Timeout is
high
(hours) and consumer crashes
Message get
duplicates
and be
processed twice
if Message Visibility Timeout is
low
(seconds)
Dead Letter Queue
(image)
Features
:
Can set
MaximumReceives
threshold of how many times a message can go back to the queue if consumer fails to process message with the Message Visibility Timeout
Message goes into a Dead Letter Queue (DLQ) after the threshold is exceeded
Good to set retention of
14 days
in the DLQ to debug the messages in DLQ before expire
Message is moved from the main SQS queue to the default DLQ
Delay Queue (image)
Features
:
Delay a message and do not let consumers to see it immediately
Up to
15 minutes
Default is
0 seconds
(message is available right away)
Can set a default at
queue
level
Can do per-message delay using
DelaySeconds
parameter
Request-Response Systems (image)
Features
:
Is a
Pattern
Use
SQS Temporary Queue Client
to implement this pattern
Leverages virtual queues instead of creating/ deleting SQS queues (Cost-effective)
Without worry about the implementation
FIFO Queue
(image)
Features
:
Ordering of messages in the queue
Pros
:
Exactly-once send capability (by removing duplicates)
Messages are processed in order by the customer
Constraint:
Limited throughout
300 message/s without batching, 3000 message/s with batching
Use
Group ID
to group related messages and send to different consumers (image)
SNS
Security
Encryption
Features
:
In-flight encryption using
HTTPS API
At-rest encryption using
AWS KMS keys
Client-side encryption if client wants to perform encryption/ decryption itself
Access Controls
IAM policies
to regulate access to SNS API
SNS Access Policies
Features
:
Useful for
cross-account access
to SNS Topics
Useful for allowing
other services
(e.g. S3) to write to SNS Topic
Features
:
Event producer only send messages to
1
SNS topic
Many Event subscribers through subscription to listen to the SNS topic notifications
Up to
10,000,000
subscriptions per topic
4,
100,000
topics limit
Event
subscribers
can be: SQS, HTTP/ HTTPS, Lambda, Emails, SMS, mobile notifications
Event
producer
can be: CloudWatch (for alarms), ASG notifications, S3 (on bucket events), CloudFormation (for state changes, failed to build)
How to publish?
Topic Publish
using SDK:
Create a Topic
Create subscription(s)
Publish to Topic
Direct Publish
for mobile apps SDK:
Create a platform application
Create platform endpoint
Publish to platform endpoint
Works with Google GCM, Apple APNS, Amazon ADM
FIFO Topic (image)
Features
:
Ordering by
Message Group ID
(all messages in the same group are ordered)
Deduplication using a
Deduplication ID
or
Content-Based Deduplication
Can only have SNS FIFO queues as subscribers
Limited throughout
Message Filtering (image)
Features
:
Use
JSON policy
to filter messages sent to SNS topic's subscriptions
Subscriber receives
all messages
if a subscription does not have a filter policy
Example: Users visit a website to place insurance quote requests. These quote requests can be for car insurance, boat insurance, or life insurance. When an insurance quote request is placed, the request is forwarded to a series of backend systems. There’s a sales system that handles vehicle insurance for cars and boats, another sales system that handles life insurance, and an analytics system that needs to process all quote requests
Kinesis
Features
:
Collect, process, analyze
streaming data
in
real-time
Ingest real-time data (e.g. application logs, metrics, website
clickstreams
, IoT telemetry data)
Kinesis Data Streams (image)
Features
:
Capture, process and store data streams
Billing is
per shard provisioned
Can have
unlimited
shards as I want
Retention in data stream between
1 (by default) to 365 days
Ability to
re-process
(replay) data
Immutability
: data cannot be deleted once it is inserted in Kinesis
Data that shares the same partition (by partition key) goes to the same shard (ordering)
(image)
Pub/Sub pattern
Streaming service for ingest at scale
By default, data records in Kinesis are only accessible for 24 hours from the time they are added to a stream
Kinesis Data Firehose (image)
Features
:
Load data streams into AWS data stores (e.g. S3/
Redshift
/ ElasticSearch/ 3rd party/ custom HTTP)
Fully managed service, no administration, automatic scaling, serverless
Pay for data going through Firehose
Near real-time. 60 seconds latency minimum for non full batches or minimum 32MB of data at a time
Supports many data formats, conversions, transformations, compression
Supports custom data transformations using AWS
Lambda
No data storage
Kinesis Data Analytics (image)
Features
:
Analyze data streams with
SQL
or Apache
Flink
Fully managed, no servers to provision
Automatic scaling
Real-time analytics
Pay for actual consumption rate
Can create streams out of the real-time queries
Use cases
:
Time-series analytics
Real-time dashboards
Real-time metrics
SNS + SQS Fan Out pattern (image)
Features
:
Push once in SNS, receive in all SQS queues that are subscribers
Have to configure SQS Queue Access Policy allows for SNS to write
Pros
:
Fully decoupled, no data loss
SQS allows for: data persistence, delayed processing and retries of work
Ability to add more SQS subscribers over time
Use cases: S3 Event to multiple queues
S3 event type (e.g. object create) and prefix (e.g. images/)
Send S3 event to multiple SQS queues
(image)
SNS FIFO + SQS FIFO Fan Out pattern (image)
Fan out + ordering + deduplication
Kinesis vs SQS ordering (image)
SQS vs SNS vs Kinesis (image)
Amazon MQ
Features
:
Managed Apache ActiveMQ
Does not scale as much as SQS/ SNS
Runs on a dedicated machine
Can run in HA with failover (image)
Has both Queue (SQS) and Topic (SNS) features
Use cases
: migrating from on-premise running application which may use open protocols (e.g. MQTT, AMQP, STOMP, Openwire, WSS) to cloud
Event Processing design pattern
:
SQS, SNS, Lambda, DLQ architecture (image)
Fan out pattern deliver to multiple SQS (image)
S3 Events (image)