From the course: Data Engineering Pipeline Management with Apache Airflow
Join today to access over 24,900 courses taught by industry experts.
Configuring DAG-level SLAs - Apache Airflow Tutorial
From the course: Data Engineering Pipeline Management with Apache Airflow
Configuring DAG-level SLAs
- [Instructor] I've updated the code for our branching pipeline here to configure an SLA for the DAG as a whole. This means that from the start of the DAG run till the end of its execution, the entire DAG should be executed within the SLA we have defined as a time delta. Once again, remember that SLAs only work for scheduled DAGs, so if your DAG has no schedule, SLAs will not work. If your DAG has a schedule and you trigger it manually, again, your SLAs will not be enforced. This has to do with how Airflow treats your scheduled DAG runs. Airflow only considers the schedule execution time of each DAG run and the interval between runs. If your DAG does not have an interval between runs, meaning there is no schedule, Airflow fails to calculate the execution dates and thus, cannot compute SLAs. Now, notice the default_args for my DAG here. On line 24, I have the SLA for the DAG, time delta of five seconds. I want this…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.