From the course: PySpark Essential Training: Introduction to Building Data Pipelines
Join today to access over 24,900 courses taught by industry experts.
Recap of key concepts and next steps - Python Tutorial
From the course: PySpark Essential Training: Introduction to Building Data Pipelines
Recap of key concepts and next steps
- Congratulations on completing this course. Here's a quick recap. You learned how Apache Spark works under the hood and how PySpark lets you harness that power using Python. We explored the data frame API, tackled common operations like filtering, joining, and aggregating, and talk through handling missing data. You saw how PySpark SQL fits into the picture, and how to make SQL and Python when it makes sense. And finally, we looked at what running PySpark in production actually looks like beyond the notebook and discussed some cloud-based options for running PySpark pipelines. Now you've got a solid foundation of PySpark. If you want to take your learning even further, here's some concepts you could explore. You might want to look into structured streaming if you're working with real-time data. Another interesting concept is Delta Lake, an open source project which helps with versioning and data consistency in production environments. And if you're interested in machine learning…
Download courses and learn on the go
Watch courses on your mobile device without an internet connection. Download courses using your iOS or Android LinkedIn Learning app.