Importance of Data Transformation

Most #datagovernance challenges stem from systems and processes that were designed without explicit data management principles. The larger, older, and more complex the organization is, the larger the problems tend to be. Organizations have tried to retroactively implement data governance for legacy systems and processes. I spent several years in the trenches of such programs, tracking down elusive system owners, and digging through outdated ETL scripts and technical metadata – it’s a grind. The approach involves extensive efforts to untangle #data flows, working with sometimes no longer supported technologies, and imposing governance standards on outdated or non-compliant systems. Engineers and consultants that were responsible in days past have long gone, further complicating efforts. Faced with mounting costs and frustration, many organizations simply give up. Savvy data leaders have begun to focus on “designing in” data governance from the start of new transformations, marking a move from remediation to prevention, and from reactive to proactive. The key to integrating data governance into the fibers of your data #architecture is to ensure that your organization’s transformation lifecycle methodology includes data governance commandments. Having an organizational transformation lifecycle ensures that any transformation of a certain impact or size is automatically brought into scope. This simplifies the task for data governance leaders, who would otherwise have to go out into the organization to identify the initiatives where data governance should be playing a role, each time then having to advocate for its inclusion – often a thankless task. The data governance considerations listed in the attached might sound heavy and time-consuming, but if done right, they can actually accelerate transformation, instead of delaying it. For example, using a data catalogue and adopting existing data products can save time finding and preparing data, and using an existing enterprise data model can prevent time being spent to create a new model from scratch. Trust me, the members of your data team will be much happier to enable new projects instead of to solve the mess caused by previous ones. Focusing on the #transformation lifecycle not only prevents the occurrence of new data governance issues but also yields a higher ROI. Designing systems and processes with data governance in mind from the outset reduces the costs and complexities of retrofitting governance later. Moreover, when data governance is embedded by design, the business users can leverage high-quality, well-governed data as soon as the transformation is complete. For more 👉 https://lnkd.in/e7nVGRGP 

‘Lift-and-shift’ #data migration is like moving house and dumping all your clutter into your new home. You wouldn't do that, so why do it with your data? I agree with my colleague Kevin M. Campbell, CEO of Syniti, part of Capgemini, whose recent article in Forbes argues that although lift-and-shift might feel like the faster route in the world of data migration, speed can come at a cost. The article breaks down some key considerations for leaders before any major transformation: - Poor data quality will derail digital and #AI initiatives - #Legacy inefficiencies reduce ROI from new platforms like S/4HANA - A true “data-first” approach can unlock AI and #analytics capabilities, not just preserve the past It’s my belief that transformation is about readiness as much as it’s about new technology, and data readiness is truly foundational. Organizations that take the time to assess, cleanse, and modernize their data upfront will be best positioned to realize the full value of their digital investments. Read more here: https://lnkd.in/eMhhXDHe

Imagine if every time you edited a document, the word processor forced you to retype everything that had been written before that edit. How expensive would that be for a company? This is exactly how data transformation works today. Each time a data engineer modifies some part of the data stack, the Cloud Data Warehouse & its transformation layer recalculates everything. What if the system were designed so that it only recalculated the metrics needed? How much less expensive would it be? Databricks partnered with Tobiko to quantify the impact. Selective recalculation delivers a 9x cost savings. Selective recalculation - only calculating what needs to be - delivers the 9x cost savings. This efficiency becomes critical as data transforms from a business asset into the business foundation itself with AI. Migrating to a new system is often expensive, but with SQLMesh’s dbt adapter, no code changes are required to the existing schema to support this cost savings. In a world where every cloud dollar counts, it’s time to stop forcing your data warehouse to rewrite “War and Peace” when all you need is the Cliff’s Notes — your CFO will thank you.