Please enable JavaScript.
Coggle requires JavaScript to display documents.
Apache AirFlow - Coggle Diagram
Apache AirFlow
Advantages
Very easy to use UI
Independent Scheduler
Pure Python
Operators
Functional Principles
Data Immutability
Idempotence
Open Source
Architecture
Components
Web Server
Scheduler
Executors/Workers
Sequencial
Local
Celery
Kubernetes
Mesos
Metastore
Connfiguration:
Variables and Connections
User Information:
roles and policies
DAG Information:
Schedule intervals, statistics
Architecture
Multi-node
Scale Up:
Increasing celery daemons
Scale Out:
Adding more nodes
Distributed Processing
High Availability
Single Node
Scale out
Master Nodes:
Web Server
Downsides
No Intuitive
Complexity with
changing Schedule Interval
No Native Windows support
Tricky CI/CD
Principles
Dynamic
Extensible
Elegant
Scalable
History
2014, Air BnB
2016, Apache
Foundation Incubator
Maxime Beauchemin
2015, Initial Release
Concepts
DAG
Scope
Default Arguments
Task
Task Instance
Task Life Cycle
No Status
Scheduled
Queued
Running
Success
Failed
Shutdown
Operators
Sensors
Hooks