From the course: Apache Airflow Essential Training
Join today to access over 24,900 courses taught by industry experts.
Creating a pipeline using the FileSensor - Apache Airflow Tutorial
From the course: Apache Airflow Essential Training
Creating a pipeline using the FileSensor
- [Instructor] Now that we understand how the file sensor works, in this demo, we'll build an interesting pipeline that'll wait for files that match a regular expression to be available, then read those files, write the data out to a database table and also write out some filtered results to another location in the file system. I'm logged into psql here, and I'm going to create a new database called, laptop_db. We'll create our tables here within this database. We'll now connect to this database using the \c command. And once this is done, we'll come back here when we want to look at the data populated in the table. Here I am in the Airflow UI. I'm in the Admin Connections view, and I'm going to add a new connection to the laptop_db that we just set up. I'm going to call this postgres connection to laptop db. Select the Postgres option here from this dropdown and the UI will change. And let's specify the other information…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.