From the course: Data Engineering Pipeline Management with Apache Airflow

Unlock this course with a free trial

Join today to access over 24,900 courses taught by industry experts.

Basic setup for the transformation pipeline

Basic setup for the transformation pipeline - Apache Airflow Tutorial

From the course: Data Engineering Pipeline Management with Apache Airflow

Start my 1-month free trial Buy for my team

Basic setup for the transformation pipeline

“

- [Lecturer] When you're running in a production environment, you won't be using the sequential executor, that's only for development, nor the local executor. That's only for executing tasks on your local machine. You'll want to run your task in a distributed environment, and for that, you'll use something like the Kubernetes Executor or the Celery Executor. Here in this demo, we'll see how you can use the Celery Executor to run your tasks in a distributed framework. Here I am in my virtual environment, and I'm going to run Airflow info and grep providers that have been installed along with my current version of Airflow. Notice that the Apache Airflow Providers Celery version 3.1.0 has already been installed. That's because when we installed Airflow, we also installed Celery. Celery is an open source distributed task queue system that allows you to manage task execution and scheduling in a distributed environment.…

Contents