Downloaded 10 times
Uploaded byCarl Anderson
Setting up Data Science for Success: The Data Layer
This document discusses setting up data science projects for success by focusing on the importance of data preparation. It notes that 76% of data scientists view data preparation as the least enjoyable part of their work. The document outlines various facets of data preparation, including collecting, understanding, cleaning, and reshaping data. It emphasizes that data quality is important and a shared responsibility across data engineering, data science, and business intelligence teams. It recommends creating a single source of truth for data through techniques like data dictionaries to define data for all teams.
More Related Content
Similar to Setting up Data Science for Success: The Data Layer
![[Redis Released]- FalkorDB - Redis + Graph Agentic Memory’s Secret Sauce](https://cdn.slidesharecdn.com/ss_thumbnails/redisreleased-falkordbslidedeck-1125-251115194922-e1c0046b-thumbnail.jpg?width=640&height=640&fit=bounds)
Setting up Data Science for Success: The Data Layer
- 2. Setting up DataScience for Success: The Data Layer
- 3.
- 5. 80% 10% 10% Data Preparation ModelingInterpretation & Presentation
- 6. 76% of datascientists view data preparation as the least enjoyable part of their work (Crowdflower / Figure8, 2016)
- 7. Josh Wills (Slack): ”I’ma data janitor. That’s the sexiest job of the 21st century.”
- 9. 80% 10% 10% AB C time
- 10. ● Collecting: searching,API, web scraping ● Understanding structure: entities, relations, foreign keys, UUIDs ● Understanding data: what fields / metrics mean, categorical variables ● Profiling: EDA, ranges, distributions ● Cleaning & normalizing: date formats, encodings, data types ● Reshaping and re-formatting data ● Filter, aggregate ● Deal with scaling issues, waiting for queries, set indexes & partitions
- 11. Various facets ofdata quality: ● Accurate: right types/format ● Coherent: referential integrity, no dupes ● Complete: no missing records & values ● Timely: in order, not late ● Defined: data dictionary, no field stuffing ● ....
- 12.
- 13. Not my Job! DataQuality has to be a shared responsibility
- 14.
- 15. ● UTC ● Stableschema ● DB constraints: no dupes ● Audit table / change log ● “Our eventing has always been ‘fire and forget’ with no guarantee of delivery.”
- 16. Validate with knownanswers
- 17. ● Monitoring /Alerts ● Retries ● Data type & range checks ● DB constraints ● Audit logs ● Schema on write
- 18. Own: ● Single Sourceof Truth ● Metric implementation ● Monitoring / Alerts ● Data Dictionary
- 19. “I found theright table and have this field but I don’t know what it means”
- 20. https://medium.com/@leapingllamas: Data Dictionary:a how to and best practices “How do we define ARPU?” “This is my understanding of active user”
- 21. ● Data qualityis important and a shared responsibility ● Create a single source of truth ● Create table-level and company-level data dictionaries Your data scientists will thank you!
- 22.