Please enable JavaScript.
Coggle requires JavaScript to display documents.
data engineering - Coggle Diagram
data engineering
what it's for/it's place
data hierarchy of needs
foundation for DS, ML, AI
analytics
business intelligence
what is is
software engineering
logging
fluentbit
locations
S3
Kibana for analysis
monitoring
prometheus
alerting
PagerDuty
AWS SNS
programming
processing, ETLs
types
batch
MapReduce
spark
stream
Apache Beam
spark
ingestion
Kafka
AWS Kinesis
serialization
avro
parquet
protobuf
json
compute
messaging
Kafka
Message Queues
RabbitMQ
visualization
Redash
D3.js
Apache Superset
Metabase
Tableau
devops
containerization
Docker
container orchestration
Kubernetes
workflow
Luigi
Airflow
specifications and standards
metrics
StatsD
Grafana for visualization
testing
functional
integration
regression
unit
system
smoke
acceptance
accessibility
non-functional
performance
load
stress
volume
security
reliability
compliance
recovery
compatiblity
backward
data
how it's accessed
distributed query engines
presto
impala
how it's used
analytics
business intelligence
data science
how it's stored
file systems
HDFS
S3
databases
RDBMS
key-value
columnar
document
graph
distributed
timeseries
other
datalakes
data warehouses
encoding
json
xml
csv
avro
parquet
MessagePack
Thrift
Protocol Buffers
what it is
facts and statistics
how it is obtained
recording, sensing
manual input
generation
inference
querying
links
https://github.com/igorbarinov/awesome-data-engineering
https://www.softwaretestinghelp.com/types-of-software-testing/