Please enable JavaScript.
Coggle requires JavaScript to display documents.
Data Engineering - Coggle Diagram
Data Engineering
Data Transformation
Data Conversions
Complex Transformations
Performance Overhead
Data Quality
Data Cleaning
Data Duplications
Quality Metrics
Data Normalization
Real Time Data Quality Assurance
Legacy Systems
Compatibility issues with current trend
ETL
Extraction Problems
Load Balancing
Version Control
Data Ingestion
Batch processing
Scheduled Jobs
Job failures
Processing Latency
ETL Pipelines
Data
Load balancing
Real time streaming
Message Queues
Queue maintanence
Data Outburst
Stream Processing
Event order
Scalability
Data Connectivity
Different sources & their different formats
Security
Data Transfer problems
Data Latency
Data ops
Monitoring & Alerting
Logging
Alert Fatigue
Monitoring Overhead
Performance optimization
Version Control
Documentation and knowledge sharing
Manual effort
Data Storage
Data Warehousing
Query Optimization
Partitioning
Data Lakes
DataSwamps
Data Bases
Schema Design
Transformation from one structure to other
Schema Evolution
Scalability
Data Gathering
Data Sources identification
Internal Databases
Data Silos
Legacy Systems
API
Rate Limits
API Changes
Web Scrapping
Website Access
Data Format
External Data providers
Data licensing
Data quality
Public Datasets
Data Accuracy
Data Timeliness
Collecting data
API
Authentication
Latency
Files
File formats
Size of the files
IOT
Data Transmission reliability on sensors
Manual Collection
Human Error
Time consuming
Data streaming
Real Time processing
Data Validation
Schema Validation
schema changes
Data Type Validation
Inconsistency
Integrity Checks
Security
Data Anonymization
Data Output
Data Visualization
Data Analysis
Insights
Data Serving
Delay