Please enable JavaScript.
Coggle requires JavaScript to display documents.
DP-203 - Chapter 11: Managing Batches and Pipelines - Coggle Diagram
DP-203 - Chapter 11: Managing Batches and Pipelines
Triggering batches
Azure Function
template: Azure Blob Storage Trigger
handling failures
error types
Pool errors
Quota limits
Insufficient resources in VNET
Timeouts
Node errors
unusable node due to start task failures
troubleshoot
GET {batchUrl}/
pools
/{poolId}/
nodes
/{nodeId}?api-version=2021-06-01.14.0
TaskFailureInformation
TaskExecutionResult
ComputeNodeError
Application download failures
Nodes going into a bad state
hardware, infra, network, ...
Job errors
troubleshoot
GET {batchUrl}/
jobs
/{jobId}?api-version=2021-06-01.14.0
JobExecutionInformation
JobSchedulingError
file output errors
Fileuploadout.txt
Fileuploaderr.txt
Task errors
troubleshoot
GET {batchUrl}/
jobs
/{jobId}/
tasks
/{taskId}?api-version=2021-06-01.14.0
TaskExecutionInformation
managing pipelines
Managing Spark jobs in a pipeline
configuring version control
Validating Batch Loads
(ADF)
Validate activity
Get metadata activity
If condition
Check for existence of file/folder
Check if schema is correct
Execute pipeline only if validation is succesful
Scheduling Data pipelines
(ADF & Synapse)
Trigger types
Scheduled
Tumbling window
Event-based
Custom