From the course: Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS

Unlock the full course today

Join today to access over 24,900 courses taught by industry experts.

Setting up your MSK cluster and EC2 instance

Setting up your MSK cluster and EC2 instance

From the course: Data Engineering Project: Build Streaming Ingestion Pipelines for Snowflake with AWS

Start my 1-month free trial Buy for my team

Setting up your MSK cluster and EC2 instance

“

- [Instructor] We'll start setting up our pipeline by building our MSK cluster and EC2 instance. The EC2 instance will serve as the data producer that sends messages to MSK. It will also host the Snowpipe Streaming via Kafka Connect. Before proceeding, pause this video here and refer to the handout for a link to a handy CloudFormation template provided by Snowflake. Click that link, but come back to this video before proceeding to kick off the build. We'll be operating out of the US-West-2 region. The CloudFormation template is set up to build in that region as well. Let's click on the View in Designer button to understand what this template will be setting up for us. We can break this JSON code down into sections. Firstly, we are setting up parameters to be referenced in the build of the EC2 instance and MSK cluster. For the MSK cluster, we are defining the subnet associations and the security group associations to ensure…

Contents