From the course: Apache Airflow Essential Training

Unlock this course with a free trial

Join today to access over 24,900 courses taught by industry experts.

Setting up a connection to Amazon S3 buckets

Setting up a connection to Amazon S3 buckets - Apache Airflow Tutorial

From the course: Apache Airflow Essential Training

Start my 1-month free trial Buy for my team

Setting up a connection to Amazon S3 buckets

“

- [Instructor] Now that we have everything set up, let's talk about what we'll be doing in our dag. We'll read data from AWS S3, using an S3 hook. And we'll write data out to a Postgres table using a Postgres hook. Here on screen, I'm logged into PSQL, this is my Postgres database. Now if you remember, we had already created a "customers_db" database. I have now connected to that database. Let's take a look at the tables here. I'll run the \dt command, and you can see that there are three tables that are present here, customers, customer purchases, and complete customer details. These are tables that were created in a previous demo. I'm going to be adding one more table to this database. I just wanted to show you that this database still exists. Let's head over to the Airflow UI. And in order to connect to S3, we need to set up a new connection. Go to Admin, Connections, and this is a page that we are familiar with.…

Contents