Download as PDF, PPTX
Uploaded byDATAVERSITY
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve Data Quality
This document discusses the importance of data observability for improving data quality. It begins with an introduction to data observability and how it works by continuously monitoring data to detect anomalies and issues. This is unlike traditional reactive approaches. Examples are then provided of how unexpected data values or volumes could negatively impact downstream processes but be resolved quicker with data observability alerts. The document emphasizes that data observability allows issues to be identified and addressed before they become costly problems. It promotes data observability as a way to proactively improve data integrity and ensure accurate, consistent data for confident decision making.
More Related Content
Similar to Keeping the Pulse of Your Data – Why You Need Data Observability to Improve Data Quality
![[Redis Released]- FalkorDB - Redis + Graph Agentic Memory’s Secret Sauce](https://cdn.slidesharecdn.com/ss_thumbnails/redisreleased-falkordbslidedeck-1125-251115194922-e1c0046b-thumbnail.jpg?width=640&height=640&fit=bounds)
Keeping the Pulse of Your Data – Why You Need Data Observability to Improve Data Quality
- 1. Keeping the Pulse ofYour Da ta : Why You Need Data Observa bility to Improve Da ta Qua lity
- 2. Spea kers Julie Skeen Sr.Product Marketing Manager Micha el Sisola k Principa l Sa les Engineer
- 3. Agenda • Introduction todata observability • How data observability works • Use case examples • Q&A 3
- 4. 47% of newly created da ta records ha ve a t lea st one critica l error 68% of orga niza tions sa y dispa ra te da ta nega tively impa cts their orga niza tion 84% of CEOs sa y tha t they a re concerned a bout the integrity of the da ta they a re ma king decisions on Precisely Da ta Trends Survey Forbes Ha rva rd Business Review Da ta integrity is a business impera tive
- 5. Introduction to Data Observa bility Business Challenges • Data downtime disrupts critical data pipelines and processes that power downstream analytics and operations • Lack of visibility around health of data reduces confidence in business decisions • Traditional manual methods do not scale, are error-prone, and are resource intensive 5
- 6. Everything old isnew a ga in • “W. Edwards Deming The Father of Quality Management” started the observability concept 100 years ago • Observability is a key foundational concept of SPC, Lean, Six Sigma and any process dependent on building quality into repetitive tasks Applying the same principles to data = data observability • Using statistical methods to control complex processes to ensure quality data products over time Wha t is Da ta Observa bility? 6 IDC; Phil Goodwin a nd Stewa rt Bond, “IDC Ma rket Gla nce: Da ta Ops, 2Q21”(June 2021) Ga rtner, Hype Cycle for Da ta Ma na gement, 2022, Melody Chien, Ankush Ja in, Robert Tha na ra j, June 30, 2022
- 7. Why Now? 7 • Businessesa re more da ta -driven tha n ever • Problema tic events a re infrequent but ca n be ca ta strophic • User’s da ta expertise ha s evolved a long with expecta tions to do more with it • Da ta prolifera tion a nd technology diversifica tion • AI ha s evolved to support the complexity of the problem
- 8. Da ta Observability is proa ctive, not rea ctive 8
- 9. Da ta Integrity and Qua lity QA is done at the time of development Ra ndom issues a re surfa ced Users find a nd report defects 9 9 Typica l Da ta Products a nd Pipelines Tra ditiona lly, the qua lity of a da ta product or pipeline is ensured during the development process a nd not throughout the opera tiona l lifecycle. Da ta Product(s) X Da ta Source #1 ? Da ta Source #2 ? Da ta Source #3 ? Da ta Source #4 ? Crea te a nd/ or Source The Da ta Tra nsform Da ta Enrich / Blend / Merge Da ta Publish a n Expose Da ta P r o c e s s
- 10. 10 10 Da ta Pipelineswith Observa bility Da ta Observa bility tools observe the performa nce of da ta products a nd processes in order to detect significa nt va ria tions before they result in the crea tion of erroneous work product in reports, a na lytics, insights a nd outcomes. Da ta Source #1 Da ta Source #2 Da ta Source #3 ! Da ta Source #4 Crea te a nd/ or Source The Da ta Tra nsform Da ta Enrich / Blend / Merge Da ta Publish a n Expose Da ta P r o c e s s Observing ea ch sta ge in the pipeline Issues identified a nd resolved prior to fina l product O b s e r v e Da ta Product(s)
- 11. 11 Da ta Observa bility Impact of Unexpected Da ta Da ta a noma lies ha ve downstrea m impa cts, but not every issue impa cts the process in the sa me wa y. The sooner you ca n detect a noma lies, the sooner you ca n a ssess the impa cts a nd effectively remedia te. EXAMPLE
- 12. How Da taObserva bility Works Discovery Ana lysis Action
- 13. Intelligent Ana lysisIdentifies Anoma lies 13 AI identifies trends tha t tra ditiona l methods ca nnot ea sily find Ra ndom Noise Upwa rd Trend Downwa rd Trend Step Cha nge 2 Step Cha nge 1 Sudden Jump Up
- 14. Da ta Observability a nd Qua lity 14 Rules Metadata Time Data Quality Management Da ta Observa bility Focused Ca pa bilities • Alerts a nd da shboa rds for overa ll da ta hea lth trending a nd threshold a na lysis • Anoma ly detection ba sed on volume, freshness, distribution a nd schema meta da ta • Predictive a na lysis simula ting huma n intelligence to identify potentia l a dverse da ta integrity events “Observa bility is the missing piece toda y to give our da ta stewa rds a ccess to da ta discovery insights without ha ving to go to IT for queries or reports” - Jea n-Pa ul Otte, CDO, Degroof Peterca m
- 15. Alerts a ndImpa cts 15 Volume Alert Impacts
- 16. Use Ca seExa mples
- 17. 17 Da ta Observa bility Impact of Unexpected Va lues An incorrect currency type in the order crea ted a n infla ted revenue a mount which would ha ve resulted in the incorrect tota l revenue a mount. The error wa s ca used beca use the currency conversion ta ble wa s not upda ted. The Da ta Observa bility solution would notify the Da ta Ops tea m of the da ta drift so tha t they could quickly resolve the issue a nd prevent it from impa cting downstrea m a na lytics a nd rela ted decisions. EXAMPLE
- 18. 18 Da ta Observa bility Unexpected data volumes impa ct opera tions A single-da y spike of 500% in the dolla r a mount of orders ca used beca use the compa ny expa nded into a new geogra phy without notifying a ll a ffected a rea s within the compa ny. Da ta stewa rd would receive a volume a lert which a llows them to quickly investiga te the issue before it impa cts downstrea m a na lytics a nd rela ted decisions. EXAMPLE
- 19. Use Ca seReca p 19 • Da ta a noma ly impa cted downstrea m processes • Impa ct of Unexpected Va lues ca used by a n inva lid currency type • Unexpected data values ca used by la ck of communica tion interna lly
- 20. Understa nd thehea lth of your data with continuous measuring and monitoring Obta in visibility into your da ta la ndsca pe a nd dependencies with intuitive self-discovery ca pa bilities Receive a lerts when outliers a nd a noma lies a re identified using a rtificia l intelligence Resolve da ta drift a nd shift when identified by intelligent a na lysis 1 2 3 4 Enable quick remediation when issues occur by understanding the cause of the issue 5 Da ta Observa bility benefits 20
- 21. Da ta Observability Proactively uncover data a noma lies a nd ta ke a ction before they become costly downstrea m issues
- 22. For trusted data , you need da ta integrity Data integrity is data with maximum a ccura cy, consistency, a nd context for confident business decision-ma king Da ta Integrity
- 23. The modular, interoperablePrecisely Data Integrity Suite conta ins everything you need to deliver a ccura te, consistent, contextua l da ta to your business - wherever a nd whenever it’s needed. 23
- 24. 7 strong modulesdeliver exceptiona l va lue Da ta Integra tion Da ta Observa bility Da ta Governa nce Da ta Qua lity Geo Addressing Spa tia l Ana lytics Da ta Enrichment Break down da ta silos by quickly building modern da ta pipelines tha t drive innova tion Proa ctively uncover da ta a noma lies a nd ta ke a ction before they become costly downstrea m issues Ma na ge da ta policy a nd processes with grea ter insight into your da ta ’s mea ning, linea ge, a nd impa ct Deliver da ta tha t’s a ccura te, consistent, a nd fit for purpose a cross opera tiona l a nd a na lytica l systems Verify, sta nda rdize, clea nse, a nd geocode a ddresses to unlock va lua ble context for more informed decision ma king Derive a nd visua lize spa tia l rela tionships hidden in your da ta to revea l critica l context for better decisions Enrich your business da ta with expertly cura ted da ta sets conta ining thousa nds of a ttributes for fa ster, confident decisions
- 25.
- 26. Tha nk you Learn more a bout Da ta Observa bility https://www.precisely.com/product/data -integrity/ precisely-da ta -integrity-suite/ da ta -observa bility