17.197: Order data into Kinesis

Kinesis

image

Using a "Partition Key" value of the "truck_id"

Same key will always go to the same shard

SQS

No ordering

For SQS FIFO, if not use Group ID, messages are consumed in the order they are sent, with only 1 consumer

You want to scale consumer number, but you want message to be "grouped" when they are related to each other

then you use a group ID (same as Partition key in Kinesis)

Example

Let assume 100 trucks, 5 kinesis shard, 1 SQS FIFO

Kinesis

20 trucks/shard

Max consumer in parallel is 5

Can receive up to 5MB/s of data

SQS FIFO

Have 100 group ID

Have up to 100 Consumers

Have up tp 300 message per second (or 3000 if using batching)