Download as PDF, PPTX
Uploaded byDatabricks
Building Data Quality pipelines with Apache Spark and Delta Lake
This document covers a presentation on building data quality pipelines using Apache Spark and Delta Lake, emphasizing the significance of addressing dirty data which costs companies significantly. The speakers outline key design decisions for creating a robust system that meets specific business needs while facilitating ease of use for developers. Conclusively, it highlights the benefits of building custom solutions over off-the-shelf products, particularly in enhancing data ingestion processes.
In this document
Powered by AI
Presentation on creating data quality pipelines using Apache Spark and Delta Lake.
Bios of speakers Sandy May and Darren Fuller, tech champions in data engineering and passionate about Spark and Azure.
Outline of the presentation covering problems, needs, solutions, investigation, and learning.
Overview of data quality problems, highlighting costs of dirty data ($3 trillion) and challenges faced by data scientists.
Discussion on whether to build or buy data quality solutions, examining ownership, features, and cost considerations.
Key design decisions for data quality solutions including cross-cloud capability and tool usage.
Concept of Enterprise Data Warehouse and its relevance in data management.
Call to action for the audience to engage in building data quality solutions.
Final recap of the presentation discussing the effectiveness of building data quality solutions.
Conclusions about the quick and effective build of data quality solutions, emphasizing business needs.
Open floor for audience questions related to the presentation.
Closing remarks encouraging feedback and reviews from the audience.
More Related Content
Similar to Building Data Quality pipelines with Apache Spark and Delta Lake
![[Redis Released]- FalkorDB - Redis + Graph Agentic Memory’s Secret Sauce](https://cdn.slidesharecdn.com/ss_thumbnails/redisreleased-falkordbslidedeck-1125-251115194922-e1c0046b-thumbnail.jpg?width=640&height=640&fit=bounds)
Building Data Quality pipelines with Apache Spark and Delta Lake
- 1. Building Data Quality pipelineswith Apache Spark and Delta Lake Sandy May & Darren Fuller Lead Data Engineers Elastacloud
- 2. Speaker Bio Sandy May- @spark-spartan Databricks Champion Data Science London Co-Organizer Tech speaker across the UK Passionate about Apache Spark, Databricks, AI, Data Security and Reporting platforms in Microsoft Azure
- 3. Speaker Bio Darren Fuller- @dazfuller Databricks Champion Tech speaker across the UK Passions include Apache Spark, Microsoft Azure, Raspberry Pi
- 4. Agenda Sandy May What isthe problem? What do we need? How can we make it easy to use? Darren Fuller How can we investigate? Where do we go from here? What have we learnt
- 5.
- 6. What is theproblem? • Harvard Business review suggested Dirty Data cost US companies $3 trillion in 2017 • Business data is hard to clean generically, it often requires domain knowledge • Dirty Data can be frustrating for Data Scientists and BI Engineers • In the worst case, Dirty Data can provide incorrect reports and predictions leading to potential significant losses @dazfuller @spark_spartan
- 7. Should we Buildor Buy? § Own the IP § Prioritise the features you want § Built for your use case § No licence fees § Use your core technology § May have track record § Bugs fixed by vendor § Features not thought about by business § Service Level Agreements Buy Build @dazfuller @spark_spartan
- 8. Key Design Decisions ▪Support to Run Cross Cloud ▪ Use Native tools in Azure and AWS ▪ Easy for SQL Devs to write rules ▪ Single Reporting Platform ▪ Capability to reuse custom business rules ▪ Run as part of our Data Ingestion Pipelines with Delta Lake @dazfuller @spark_spartan
- 9. Enterprise Data Warehouse @dazfuller@spark_spartan
- 10. Let’s Build it! @dazfuller@spark_spartan
- 11.
- 12. Conclusions ▪ Building canbe quick and effective ▪ Prioritise your own business needs, you know your data best ▪ Can be used as a stop gap while you create a service for an off the shelf product ▪ Easy to run as part of ingestion pipelines ▪ Business value in reports and reuse of rules ▪ Use Delta Lake for Schema Evolution @dazfuller @spark_spartan
- 13. Thanks for listening! Questions? @dazfuller@spark_spartan
- 14. Feedback Your feedback isimportant to us. Don’t forget to rate and review the sessions.