From the course: Apache Airflow Essential Training
Join today to access over 24,900 courses taught by industry experts.
Running DAGs with a FileSensor - Apache Airflow Tutorial
From the course: Apache Airflow Essential Training
Running DAGs with a FileSensor
- [Instructor] Now, before we run our DAG, notice here on top I have the tmp folder which is completely empty. I have a number of files with the laptops prefix here in my datasets folder below. Now, in the real world, it's possible that your data is uploaded to a certain location, say once a day or once every hour and you might want your DAG to run every hour, process this data, and write out some output. That's what we're mimicking here in this demo. Initially we'll have no data, and then as the data comes in we'll trigger our DAG to process that data. Now I've gone directly to the graph view for my DAG. This is the pipeline with the file sensor, and you can see what the dependencies look like. Create table then checking for the file, then inserting the laptop data, then performing the three filtering operations, and then finally, deleting the original files. Now let's go ahead and run this DAG. I'm going to unpause this…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.