Please enable JavaScript.
Coggle requires JavaScript to display documents.
Google Cloud Composer (features (environments (cloud storage (google cloud…
Google Cloud Composer
features
management
web interface
CLI
configuration
access control
logging and monitoring
environments
web server
database
cloud storage
google cloud bucket
DAG
plugins
data
logs
Airflow Concepts
workflow
DAG
Task
Task instance
subDAG
Trigger Rules:
all_success, all_failed, all_done and etc
Latest run only
packaged DAGs
task state
running
zomibe
undead
scope
local
global
default arguments
context manager
hooks
: keep authentication code and information out of pipelines
pools
: limit the execution parallelism
connections
: related to hooks
XComs: exchange msgs between tasks
cluster policy
documentation & notes
: visible in web interface
Google Cloud Composer
Architecture
Tenant Project
Airflow Database
(Cloud SQL): 1. stores metadata 2. Access control (Cloud SQL proxy in GKE cluster)
Airflow Web Servers
(App Engine Flex)
Identity-Aware Proxy
Customer Project
(deployed in GKE)
Airflow Worker
Redis
Airflow Scheduler
CeleryExecutor
Cloud Storage
DAGs
Logs
Plugins
Logs
(Stackdriver)
Airflow Operator
by function
BashOperator
gloud
bq
gsutil
kubectl
PythonOperator
google-cloud-bigquery
google-cloud-dataflow
EmailOperator
Google Cloud Platform Operators
Cloud Dataflow operators
Cloud Dataproc operators
Cloud Datastore operators
Cloud Machine Learning Engine operators
Cloud Storage operators
KubernetesPodOperator:
link
GKEPodOperator:
network enabled
by type
sensor
transfer: move data
action
by source
community-contirbuted
core
API reference:
https://airflow.incubator.apache.org/code.html#operators
Cloud Composer Operator
https://cloud.google.com/composer/docs/how-to/using/using-kubernetes-pod-operator
Airflow + Docker
kube-airflow
https://github.com/mumoshu/kube-airflow
puckel/docker-airflow
https://github.com/puckel/docker-airflow
gs://bucket-name/
Google Cloud Composer overview:
https://cloud.google.com/composer/docs/concepts/overview
architecture diagram:
https://cloud.google.com/composer/docs/images/architecture.svg
Airflow Concepts
https://airflow.apache.org/concepts.html
release notes:
https://cloud.google.com/composer/docs/release-notes
Current Airflow Version as of Oct 24, 2018:
composer-1.3.0-airflow-1.9.0
what does network enabled mean?
