From the course: End-to-End Real-World Data Engineering Project with Databricks
Join today to access over 24,900 courses taught by industry experts.
Databricks project architecture diagram
From the course: End-to-End Real-World Data Engineering Project with Databricks
Databricks project architecture diagram
- [Instructor] The high-level architecture diagram for our global retail Databricks solution looks something like this. Here, you can see that our customer data is coming from CRM, and we are keeping it in our DBFS location. Similarly, the product data is coming from inventory, is going to get stored at DBFS, and transaction data from our point-of-sale, and online store is coming into a DBFS location. In this project, we are not using any cloud. If we were using any cloud, maybe like Azure or AWS, then this DBFS location could be of, and Azure Delta Lake storage, or could be Amazon S3. But just to keep the things simple, we have, putting all that data together in the DBFS, that is Databricks file system only. So we assume that, our upstream system is going to push the data into this DBFS folder locations. Now, once the data is in DBFS, we are going to create three database layers, bronze layer, silver layer, and gold layer. These are three different databases, and data move from…
Practice while you learn with exercise files
Download the files the instructor uses to teach the course. Follow along and learn by watching, listening and practicing.
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.