Please enable JavaScript.
Coggle requires JavaScript to display documents.
THROTTLING - Coggle Diagram
THROTTLING
HYBRID
devops: multiple adb_fw templates
job prioritization
LIST OF TUPLES=[(JOB_ID,JOB_PRIORITY_LEVEL)]
dynamic MAX CEILING
SPARK CLUSTER USAGE QUERY AT BEGINNING
dynamic thread limit+QUEUE
devops: breaking delta table job run into multiple templated framework runs
adb_fw_002
ETL: 51-100
adb_fw_003
ETL: 101-150
adb_fw_001
ETL: 0-50
job throttling
DELAY SOME or ALL JOBS
intermittent SLEEP
thread throttling
limit total # of threads to MAX_THREAD or MAX_PROC CNT
multiple jobs per thread
1 job per thread
QUEUE additional jobs if # of JOBS > # MAX_THREAD/PROC CNT!
job prioritization via JOB LEVEL: higher level #==>higher priority
ID_LIST=[(1,L#),(2,L3),(3,L#)]
dynamic limit: check system load and adjust MAX RESOURCES