From the course: Data Engineering Pipeline Management with Apache Airflow

Unlock this course with a free trial

Join today to access over 24,900 courses taught by industry experts.

Executing a simple SQL DAG

Executing a simple SQL DAG - Apache Airflow Tutorial

From the course: Data Engineering Pipeline Management with Apache Airflow

Start my 1-month free trial Buy for my team

Executing a simple SQL DAG

“

- [Instructor] Before we move on, let's look at the second DAG that I have set up. This is a simple SQL pipeline that creates a table in the SQLite database and inserts some data into that table and performs a few operations. Here is a definition of my DAG, the simple_sql_pipeline. I have a create_table SqliteOperator, which creates the user's table if it doesn't already exist. Notice that this SqliteOperator uses the sqlite_conn_id, my_sqlite_conn. All of our operators will use the same connection id. Then if you scroll down, you'll see I have an insert_values_1 SqliteOperator which inserts some user data into the SQLite table. I have an insert_values_2 SqliteOperator which inserts some more data, four more records into the SQLite table. Scrolling down further, I have a SQLite operator which simply performs a SELECT * FROM users. In order to view this result, I have set do_xcom_push to True. This means this result…

Contents