From the course: Data Engineering Pipeline Management with Apache Airflow
Join today to access over 24,900 courses taught by industry experts.
Summary and next steps - Apache Airflow Tutorial
From the course: Data Engineering Pipeline Management with Apache Airflow
Summary and next steps
- [Instructor] And with this demo, we come to the very end of this course on Data Engineering Pipeline Management with Apache Airflow. Let's take a quick look at what we covered in this course. We first used role-based access control to manage user access to the Airflow UI. We saw the permissions associated with the default rules that Airflow provides. We also created our own custom role. We then worked with SLA Management in Apache Airflow. We saw how we could track SLA breaches and configure email alerts for SLA misses. We then saw how data-aware scheduling works with datasets. We created producer pipelines that updated datasets and consumer pipelines that consumed these datasets and were scheduled once the datasets that it was waiting on were available. Then we used plugin to configure Airflow with custom features, and we saw how we could build a CSV viewer right into our Airflow UI. And finally, we scaled the execution…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.