Data Contracts Management: Schema Registry and Beyond

Data Contracts Management: Schema Registry and Beyond

• 1.
  Data Contracts Management SchemaRegistry and Beyond
• 2.
  About me Pietro LaTorre Strategy & Business Development @ Quantyca SpA Quantyca is a privately owned technological consulting firm specialized in data and metadata management based in Italy quantyca.it
• 3.
  Always a goodidea Data contracts Data as a Product Data Centricity Data Fabric Data Contract Needed toformalize applications’ accountability on generated data Foundational to proactively provide metadatathat enables automation Essential to clarify meaning, shape, content and how to access data images: Flaticon.com Formal agreement between data producer and consumers on the structure and semantics of data exchanged
• 4.
  Key characteristics Good datacontracts Addressable Expressive Scoped Stable Reliable Computable Data Contract images: Flaticon.com
• 5.
  From an architecturalperspective An interesting paradox images: Flaticon.com Slow and ruled context with vulnerable flows Fluid landscape resilient to changes
• 6.
  Schema is notenough Data Contracts How do I use this data? Where can I consume it? How? How often it is updated? Who is allowed to use it? What is its quality? What does this value mean? API How-to for data interactions • Service location • Communication protocols • Authentication methods • Endpoints • Schema • SLO • SLA • Terms and conditions • Deprecation policy • Billing policy • Expected usage patterns • Schema tagging • Schema linking • Semantic linking Constraints Rules to properly use data Semantic Context and meaning
• 7.
  A socio-technical puzzle Adoptionchallenges images: Flaticon.com Define and enforce Specifications and tools still immature Require evolution of operating model, strong sponsorship and engagement Technical perspective Organizational perspective Consumer Producer Negotiate and own Team A Team B Data Contract
• 8.
  Standardization through aspecification Addressing the technical challenge dpds.opendatamesh.org Automation Interoperability Scalability images: Flaticon.com
• 9.
  Validation and Monitoring EnforcingData Contracts CEP Contract Enforcing Point Central governance of data contracts’ lifecycle CDP Contract Decision Point Integration with underlying tech stack CIP Contract Information Point Census for archival and retrieval CEP Data Contract CDP CDP CDP 1 CIP 4 2 3 5 Validation CEP CDP CDP CDP CIP 1 3 4 Monitoring 2
• 10.
  A unified approachacross service and platforms Mastering Data Contracts • Data contracts span across numerous services and platforms • Schema Registry serves as a cornerstone • Applying a holistic strategy ensures end-to-end effectiveness Data product Data contract CDP CDP CIP Confluent Platform Schema Registry CEP Data Platform Data Contracts Control Center
• 11.
  Thank you. @pietrolatorre pietro.latorre@quantyca.it PietroLa Torre