From the course: Cloud Hadoop: Scaling Apache Spark

Unlock the full course today

Join today to access over 24,900 courses taught by industry experts.

Architecture: Streaming for prediction

Architecture: Streaming for prediction - Apache Spark Tutorial

From the course: Cloud Hadoop: Scaling Apache Spark

Start my 1-month free trial Buy for my team

Architecture: Streaming for prediction

“

- We've covered quite a few capabilities of Spark in the previous movies. Let's pause now and consider how we would use them in a business scenario. So, let's look in architecture for streaming. Now, the used case here is for dynamic price prediction. So the idea is you're running like a travel site or hotel site and you want prices to be predicted based on data that's coming in a stream. So this architecture's going to use Databricks on Amazon in this case. It's going to use the data lake architecture, so that's going to be Amazon S3. It's going to use streaming service. It's also going to use machine learning. So, this shows an example of where you can go in a real world pulling these technologies together. So let's first look at the data inflow for our dynamic price prediction. We're going to address that incoming information, the messages, using the Amazon Kinesis service. Now this is a server less service that's…

Contents