ADF's Copy Activity Won’t Save You From Source Changes, Here’s Why You Need Schema Drift Handling
If you’ve built data pipelines in Azure Data Factory, you know this:
Copy Activity works great when your source schema is fixed.
But what happens when…
- A column is added?
- A data type changes?
- A column gets renamed or dropped?
Your pipeline doesn’t break immediately but your data does.
You’ll start missing columns in your destination
Data mapping mismatches
Silent failures that corrupt your data lake
How I handle this in real-world projects:
1. Enable Schema Drift in Data Flows especially when working with semi-structured or CSV data
2. Always use Mapping Data Flows with dynamic column handling
3. Log your source metadata before ingestion to track unexpected changes over time
4. Set alerts on Copy Activity’s output schema mismatch
Real lesson:
Cloud pipelines don’t fail loudly they fail quietly when you ignore schema drifts.
Plan for schema flexibility BEFORE it hits production.
#DataEngineering#AzureDataFactory#AzureDataLake#SchemaMismatch#DataFlow#MappingDataFlow
Hello Everyone,
Infosys Interview Question : Azure Data Factory Scenario #2
Q:You are ingesting files from multiple vendors into Azure Data Lake using Azure Data Factory.
Sometimes, vendors add or remove columns in their CSV files.
This causes schema mismatch errors, and the pipelines fail.
👉 How would you handle schema drift in ADF to keep your pipelines running smoothly?
Answer (detailed):
This is a real-world challenge every Data Engineer faces.
When your data source changes unexpectedly — new column added, one column dropped your ADF pipeline might stop, leaving you with a 2 AM “Pipeline failed” notification .
Here’s how I’d solve it 👇
1️⃣ Enable Schema Drift in Mapping Data Flow
• In your Mapping Data Flow, turn on Allow schema drift.
• This ensures ADF automatically accommodates new or missing columns instead of failing.
• Use Auto Mapping to let all detected fields flow through dynamically.
2️⃣ Land Raw Data in Bronze Layer (ADLS)
• Always store source files “as-is” in a Bronze Layer.
• Use Parquet or Delta formats to preserve evolving schema structures.
• Add metadata fields like ingestion_time, source_name, and schema_version for tracking.
3️⃣ Maintain a Schema Registry / Metadata Table
• Create a table (in SQL/Delta) to track your expected schema for each dataset.
• Each new load compares actual schema vs expected schema.
• Log any mismatches and alert your team automatically.
4️⃣ Handle Drift Gracefully
• Missing columns → fill with NULL or default values.
• New columns → store temporarily in an “extras” JSON field until confirmed and modeled.
• This ensures no data loss, even during schema changes.
5️⃣ Add Monitoring & Alerts
• Integrate Logic Apps or Azure Monitor for notifications when drift occurs.
• You’ll get an email/slack alert before a user notices a problem downstreams.
#Azure#DataEngineering#DataFactory#Infosys#InterviewPreparation#BigData
❓ 𝗪𝗵𝘆 𝗱𝗼 𝘀𝗼 𝗺𝗮𝗻𝘆 𝗔𝗜 𝗶𝗻𝗶𝘁𝗶𝗮𝘁𝗶𝘃𝗲𝘀 𝗳𝗮𝗶𝗹 𝘁𝗼 𝘀𝗰𝗮𝗹𝗲?
Because they skip the fundamentals. Without trustworthy, well-governed, and discoverable data, even the best AI models struggle to deliver consistent value. That’s why every organisation needs a clear, structured framework.
❓ 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗗𝗮𝘁𝗮 𝗧𝗿𝗶𝗻𝗶𝘁𝘆™ 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸?
It’s a three-layer model designed to help organisations unlock the full value of their data and AI initiatives by building step-by-step capability:
Foundational Layer
Focus on data quality, governance, access, and compliance. Create trust.
Semantic Layer
Introduce shared understanding through metadata, ontologies, and knowledge graphs.
Conversational Layer
Enable everyone to interact with data using natural language and intelligent AI interfaces.
❓ 𝗪𝗵𝘆 𝘀𝗵𝗼𝘂𝗹𝗱 𝘆𝗼𝘂𝗿 𝗼𝗿𝗴𝗮𝗻𝗶𝘀𝗮𝘁𝗶𝗼𝗻 𝗮𝗱𝗼𝗽𝘁 𝗶𝘁?
✅ It reduces duplication of effort by up to 40%
✅ Accelerates data product delivery by 3x
✅ Bridges the gap between technical teams and business users
✅ Enables true self-service, driven by trust and shared language
❓ 𝗪𝗵𝗮𝘁’𝘀 𝘁𝗵𝗲 𝗲𝗻𝗱 𝗴𝗼𝗮𝗹?
A truly data-literate, AI-enabled organisation - where every person can find, understand, and use data effortlessly.
Data silos aren’t just a tech problem - they’re an operational bottleneck that slows decision - making, erodes trust, and wastes millions in duplicated efforts.
But we’ve seen companies like Autodesk, Nasdaq, Porto, and North break free by shifting how they approach ownership, governance, and discovery.
Here’s the 6-part framework that consistently works:
1️⃣ Empower domains with a Data Center of Excellence. Teams take ownership of their data, while a central group ensures governance and shared tooling.
2️⃣ Establish a clear governance structure. Data isn’t just dumped into a warehouse—it’s owned, documented, and accessible with clear accountability.
3️⃣ Build trust through standards. Consistent naming, documentation, and validation ensure teams don’t waste time second-guessing their reports.
4️⃣ Create a unified discovery layer. A single “Google for your data” makes it easy for teams to find, understand, and use the right datasets instantly.
5️⃣ Implement automated governance. Policies aren’t just slides in a deck—they’re enforced through automation, scaling governance without manual overhead.
6️⃣ Connect tools and processes. When governance, discovery, and workflows are seamlessly integrated, data flows instead of getting stuck in silos.
We’ve seen this transform data cultures - reducing wasted effort, increasing trust, and unlocking real business value.
So if your team is still struggling to find and trust data, what’s stopping you from fixing it?
Safeguarding information while enabling collaboration requires methods that respect privacy, ensure accuracy, and sustain trust. Privacy-Enhancing Technologies create conditions where data becomes useful without being exposed, aligning innovation with responsibility.
When companies exchange sensitive information, the tension between insight and confidentiality becomes evident.
Cryptographic PETs apply advanced encryption that allows data to be analyzed securely, while distributed approaches such as federated learning ensure that knowledge can be shared without revealing raw information.
The practical benefits are visible in sectors such as banking, healthcare, supply chains, and retail, where secure sharing strengthens operational efficiency and trust.
At the same time, adoption requires balancing privacy, accuracy, performance, and costs, which makes strategic choices essential.
A thoughtful approach begins with mapping sensitive data, selecting the appropriate PETs, and aligning them with governance and compliance frameworks.
This is where technological innovation meets organizational responsibility, creating the foundation for trusted collaboration.
#PrivacyEnhancingTechnologies#DataSharing#DigitalTrust#Cybersecurity
𝐇𝐨𝐰 𝐃𝐚𝐭𝐚 𝐄𝐧𝐠𝐢𝐧𝐞𝐞𝐫𝐬 𝐜𝐚𝐧 𝐩𝐫𝐨𝐚𝐜𝐭𝐢𝐯𝐞𝐥𝐲 𝐦𝐚𝐧𝐚𝐠𝐞 𝐬𝐜𝐡𝐞𝐦𝐚 𝐜𝐡𝐚𝐧𝐠𝐞𝐬 𝐭𝐨 𝐦𝐢𝐧𝐢𝐦𝐢𝐳𝐞 𝐝𝐢𝐬𝐫𝐮𝐩𝐭𝐢𝐨𝐧𝐬 𝐭𝐨 𝐩𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐨𝐧 𝐚𝐧𝐚𝐥𝐲𝐭𝐢𝐜𝐬 𝐬𝐲𝐬𝐭𝐞𝐦𝐬.
Schema changes in upstream databases can cause unforeseen downtime in analytical and reporting workloads. This is often a byproduct of data teams being disconnected from core engineering teams. However, your team can establish a schema change control strategy to avoid this downtime.
Key strategies include:
𝐈𝐦𝐩𝐥𝐞𝐦𝐞𝐧𝐭 𝐜𝐡𝐚𝐧𝐠𝐞 𝐝𝐚𝐭𝐚 𝐜𝐚𝐩𝐭𝐮𝐫𝐞 (𝐂𝐃𝐂) 𝐭𝐨 𝐦𝐨𝐧𝐢𝐭𝐨𝐫 𝐃𝐃𝐋 𝐜𝐡𝐚𝐧𝐠𝐞𝐬
Use CDC tools to track DDL changes across database instances
Set up alerts for schema drift between environments. Simple 'Add Table' DDL can often be propagated with no impact, but dropping or changing columns, changing keys, or partitioning logic can have impacts to production analytical workloads.
𝐌𝐨𝐧𝐢𝐭𝐨𝐫 𝐝𝐚𝐭𝐚𝐛𝐚𝐬𝐞 𝐯𝐞𝐫𝐬𝐢𝐨𝐧 𝐜𝐨𝐧𝐭𝐫𝐨𝐥 𝐬𝐲𝐬𝐭𝐞𝐦𝐬
Subscribe to repository notifications for database-related changes.
Review pull requests impacting data models and table structures.
𝐄𝐬𝐭𝐚𝐛𝐥𝐢𝐬𝐡 𝐬𝐜𝐡𝐞𝐦𝐚 𝐫𝐞𝐯𝐢𝐞𝐰 𝐩𝐫𝐨𝐜𝐞𝐬𝐬𝐞𝐬
Require peer review of proposed schema changes. In many cases schema changes require some design changes to your transformation and modeling logic.
Assess potential impacts on existing ETL processes and dashboards and plan for the schema changes before they go to production.
𝐃𝐞𝐬𝐢𝐠𝐧 𝐩𝐢𝐩𝐞𝐥𝐢𝐧𝐞𝐬 𝐭𝐨 𝐡𝐚𝐧𝐝𝐥𝐞 𝐛𝐚𝐜𝐤𝐟𝐢𝐥𝐥𝐢𝐧𝐠 𝐨𝐟 𝐝𝐚𝐭𝐚 𝐮𝐧𝐝𝐞𝐫 𝐧𝐞𝐰 𝐬𝐜𝐡𝐞𝐦𝐚𝐬
Implement temporary dual-write periods during transitions
By adopting these practices, data engineering teams can maintain analytics system stability while accommodating necessary schema changes.
#dataengineering#changedatacapture#apachekafka
I’ve lost count of projects that shipped gorgeous features but relied on messy data assets. The cost always surfaces later when inevitable firefights, expensive backfills, and credibility hits to the data team occur. This is a major reason why I argue we need to incentivize SWEs to treat data as a first-class citizen before they merge code.
Here are five ways you can help SWEs make this happen:
1. Treat data as code, not exhaust
Data is produced by code (regardless of whether you are the 1st party producer or ingesting from a 3rd party). Many software engineers have minimal visibility into how their logs are used (even the business-critical ones), so you need to make it easy for them to understand their impact.
2. Automate validation at commit time
Data contracts enable checks during the CI/CD process when a data asset changes. A failing test should block the merge just like any unit test. Developers receive instant feedback instead of hearing their data team complain about the hundredth data issue with minimal context.
3. Challenge the "move fast and break things" mantra
Traditional approaches often postpone quality and governance until after deployment, as shipping fast feels safer than debating data schemas at the outset. Instead, early negotiation shrinks rework, speeds onboarding, and keeps your pipeline clean when the feature's scope changes six months in. Having a data perspective when creating product requirement documents can be a huge unlock!
4. Embed quality checks into your pipeline
Track DQ metrics such as null ratios, referential breaks, and out-of-range values on trend dashboards. Observability tools are great for this, but even a set of SQL queries that are triggered can provide value.
5. Don't boil the ocean; Focus on protecting tier 1 data assets first
Your most critical but volatile data asset is your top candidate to try these approaches. Ideally, there should be meaningful change as your product or service evolves, but that change can lead to chaos. Making a case for mitigating risk for critical components is an effective way to make SWEs want to pay attention.
If you want to fix a broken system, you start at the source of the problem and work your way forward. Not doing this is why so many data teams I talk to feel stuck.
What’s one step your team can take to move data quality closer to SWEs?
#data#swe#ai
How do you get network data out of people’s heads — and into a single source of truth?
In most telcos, critical network knowledge still lives inside the heads of a few senior engineers, planners, or ops staff.
This "tribal knowledge" isn’t just a risk — it’s a barrier to scale, automation, and resilience.
Here’s a proven 8-step approach we use to help operators shift from siloed knowledge to a clean, trusted, Single Point of Truth (SPOT) system:
1. Identify the knowledge holders.
- Start with those who “just know” how the network works — long-serving engineers, planners, response leads.
- Map who holds what knowledge and which systems lack it.
2. Locate the ‘known unknowns’
- Which areas are poorly documented?
- Think: fibre routes, logical/physical mapping, patch panels, naming quirks.
- Prioritise gaps that create the most risk or delay.
3. Run structured interviews and walkthroughs
- Ask focused, practical questions:
“Where do you look when this fails?” “How do you trace that link?”
- Document real-world setups and legacy workarounds.
4. Standardise the input process
- Use structured templates or forms that align with your SPOT schema.
- Avoid inconsistent, free-form data collection.
5. Load, clean, and validate
- Use scripts or reconciliation tools to catch
· Duplicates
· Conflicts
· Missing fields
- Turn human knowledge into structured data.
6. Show value immediately
- Create quick wins:
· Clean circuit maps
· Accurate asset views
· Easier troubleshooting
- Involve original staff to validate and build trust.
7. Build a feedback loop
- Assign data stewards. Make updates easy (QR code at site? Slack bot? Mobile form?).
- Reconcile and improve continuously.
8. Change the culture
“If it’s not in the SPOT, it doesn’t exist.”
- Leadership must drive this shift:
· Reward documentation
· Link data quality to performance
- Make documented truth the only accepted truth.
If you’re struggling to build trust in your network data, we’ve built a practical course that helps teams clean, align, and take control — fast.
“The Hidden Cost of Bad Network Data”
📩 Message me here on LinkedIn to learn more or enrol.
To build a solid Data Foundation for AI Transformation, enterprises must ensure that data is not only available, but trusted, well-governed, and ready for intelligent use. A strong data foundation bridges the gap between business goals and AI model performance.
Below are the main components:
🔷 1. Data Strategy & Governance
- Data Ownership & Stewardship: Clear roles for who owns, curates, and validates data.
- Data Policies: Governance policies for access, usage, privacy, and compliance (e.g. GDPR, HIPAA).
- Master & Reference Data Management: Ensure consistency of critical data entities across systems.
🔷 2. Data Quality & Trust
- Data Profiling & Cleansing: Remove duplicates, fix inconsistencies, fill gaps.
- Validation Rules & Anomaly Detection: Detect data drift or broken pipelines early.
- Lineage & Provenance: Know where data comes from and how it has changed.
🔷 3. Data Architecture & Infrastructure
- Modern Data Platforms: Data lakes, warehouses, lakehouses, or vector databases.
- Real-Time vs Batch Processing: Support both operational and analytical workloads.
- Data Integration & APIs: ETL/ELT pipelines, connectors, and API-based data access.
🔷 4. Security, Privacy & Compliance
- Data De-identification & Masking: Protect PII while preserving utility.
- Role-Based Access Control (RBAC): Ensure only the right users/systems can access the right data.
- Audit Trails & Monitoring: Track who accessed what, when, and why.
🔷 5. AI-Ready Data Practices
- Labeling & Annotation Workflows: For supervised learning and fine-tuning.
- Feature Stores & Embeddings: Reusable, standardized inputs for ML/AI models.
- RAG-Enabling Structures: Chunked, semantically enriched documents for Retrieval-Augmented Generation.
🔷 6. DataOps & Automation
- CI/CD for Data Pipelines: Automate testing and deployment of data workflows.
- Metadata Management & Catalogs: Enable discovery and governance at scale.
- Monitoring & Alerting: Real-time health checks on data pipelines and quality metrics.
🔧 Personal Tip: Build Talent Across Data and Infrastructure
One of the most underestimated success factors in AI transformation?
A team that understands both the data science and the engineering foundations beneath it.
Many organizations invest heavily in AI skills, but neglect the cloud, DevOps, and data infrastructure expertise needed to scale those models in production.
To make AI real, you need:
- Data engineers who can build resilient, governed pipelines
- Platform and cloud architects who can support scalable, secure compute
- MLOps specialists who bridge model lifecycle with infrastructure operations
📌 AI doesn't run in notebooks—it runs on architecture.
And that architecture has to be designed with security, performance, and cost in mind from day one.
#AITransformation#DataEngineering#DataManagement#ArtificalIntelligence
Your ETL Job Just Failed… Now What?
Let’s talk about schema drift!
It's the silent killer of automated data pipelines.
#DataEngineers What’s your go-to strategy for managing schema drift?
Imagine this: You’re pulling data from an external API. Overnight, the schema changes.
Your ETL job?
🚨 Dead in the water.
How do you handle it?
Here’s a strategy I’ve used in real-world pipelines:
1️⃣ Use your ETL (Spark, pandas, Polars, etc.) to dynamically infer the source schema.
2️⃣ Compare it against the destination schema.
3️⃣ If it matches? Push the data.
4️⃣ If it doesn’t match:
Load only the columns that align
Alert the team that a schema change has occurred.
Want to go a step further?
Send the unmatched fields (plus a primary key) to a quarantine table for later evaluation.
✅ Reports stay live.
✅ Business doesn’t miss a beat.
✅ Your team gets time to adapt.
And here's the kicker:
🔑 Some fields are critical.
If they disappear, you need an emergency alert ⛔ ...
not just an email that gets ignored.
Build that logic in!
Schema drift is inevitable.
Your pipelines should fail safely, not fatally!
This is the kind of resilient pipeline design I build with clients at Gambill Data
If you're facing schema drift or reliability issues, feel free to reach out!
Always happy to chat!
#DataEngineering#ETL#SchemaDrift#DataOps#DataPipeline#Alerting#GambillData