Importance of Continuous Monitoring in MLOps

90% of AI models degrade over time if not monitored/maintained. QB Labs developed Coda to automate model monitoring and enhance value post-deployment. Coda is supported by the LiveOps team at QuantumBlack, AI by McKinsey, which provides long-term monitoring support ensure sustained impact from deployed models and AI-driven solutions. Rewired: The McKinsey Guide to Outcompeting in the Age of Digital and AI (Wiley, June 2023) describes AI/ML models as “living organisms” that change with underlying data. As such, it says, “they require constant monitoring, retraining and debiasing — a challenge with even a few ML models but simply overwhelming with hundreds of them.” Why is model maintenance a challenge? i) The world is constantly changing, and AI models are not isolated entities. Changes to the data they use, or to other integrated systems, have a knock-on effect. ii) New skills are needed. The skillsets required to support ongoing operations are different to those needed in the build phase of an AI project. iii) Tools for ongoing operations are different from those used to build models. The tools to sustain impact are not readily available off-the-shelf, which means projects often require custom implementations. Coda and the LiveOps team have 4 main focus areas: 1) A central dashboard to streamline workflows and offer visibility on all model maintenance operations, centralizing all actionable alerts 2) Troubleshooting, monitoring, and model retraining to sustain business impact and keep models relevant 3) Bridging capability gaps so teams can sustain optimal performance even if they lack some of the skillsets needed to do so independently 4) Diagnosis and intervention with automated, configurable analyses to build resolution plans, enabling rapid issue fixes + increased model stability. Great work by the product team driving this Andrew Ferris, Ben Horsburgh, Rohit Godha

AI readiness evaluation is NOT a one-time pre-launch checklist. It's an ongoing process that evolves alongside your AI initiatives. As AI technologies and regulations evolve swiftly, your readiness assessment must keep pace. The need for continual monitoring cannot be emphasized enough. For instance, if your predictive hiring algorithm begins to display gender bias, the speed of your response becomes paramount. Do you possess the mechanisms to identify and address this bias? Once detected, what is your team's timeline for rectifying it? These critical aspects demand careful consideration before launching that promising AI solution.

Let's talk about another buzzword... Observability 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆? AI Observability goes beyond traditional monitoring, offering a holistic view into the behavior, data, and performance of AI/ML models throughout their lifecycle. It enables precise root cause analysis, proactive issue detection, and helps build more reliable, responsible models. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗠𝗟 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴? While ML monitoring focuses on tracking performance metrics, AI observability provides a broader perspective that includes insights into: - Model behavior and performance: Understanding how models function over time, including the identification of anomalies and the analysis of performance metrics. - Data quality issues: Detecting issues such as data skew, which occurs when training/tuning data does not accurately represent live data due to mismatches, dependencies, or changes in upstream data. - Model degradation: Addressing model staleness caused by shifts in external conditions such as changes in consumer behavior or economic environments. - Feedback systems: Identifying and mitigating the impact of biased, inaccurate, or corrupted data that can degrade model quality over time. 𝗪𝗵𝘆 𝗶𝘀 𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗘𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹? AI observability provides visibility into the entire AI pipeline, enabling better decision-making around model management, such as: - Deploying new models under A/B testing: Testing new models alongside existing ones to compare performance. - Replacing outdated models: Updating models to ensure they remain effective in changing conditions. - Tweaking operational models: Making minor adjustments to improve performance based on ongoing insights. When deploying AI use cases in an enterprise... guesswork isn't an option. Trust your AI to deliver — because you’re watching its every move 👀

GenAI is hitting production across the board! 🚀 Based on dozens of conversation with data science teams over the last few weeks, the trend is clear - We’re entering a transformative phase where putting GenAI applications in production is top of the mind. Sharing three key highlights : 𝗥𝗲𝗮𝗹-𝗧𝗶𝗺𝗲 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻𝘀 & 𝗚𝘂𝗮𝗿𝗱𝗿𝗮𝗶𝗹𝘀: Continuous monitoring has become the backbone of AI deployment, providing dynamic assurance, safety, and bias management. This keeps outputs reliable and adaptable in ever-changing environments. 𝗔𝗴𝗲𝗻𝘁 𝗪𝗼𝗿𝗸𝗳𝗹𝗼𝘄𝘀 𝗢𝘂𝘁𝗽𝗲𝗿𝗳𝗼𝗿𝗺𝗶𝗻𝗴 𝗧𝗿𝗮𝗱𝗶𝘁𝗶𝗼𝗻𝗮𝗹 𝗟𝗟𝗠𝘀: We’re moving beyond isolated models to optimized workflows. These agent-based systems excel remarkably in tasks like code generation (coding copilots) and process automation (RPA). 𝗦𝗟𝗠 & 𝗙𝗶𝗻𝗲-𝗧𝘂𝗻𝗶𝗻𝗴 𝗳𝗼𝗿 𝗖𝗼𝘀𝘁 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆: Task specific small models (SLs) and fine-tuning are the preferred choice in production. They balance cost and performance, ensuring tailored and scalable AI solutions. These shifts are redefining what’s possible, setting a new benchmark for AI excellence in production. #GenAI #AIinProduction #AIDeployment #AgentWorkflows #LLMs #RealTimeMonitoring #DataScience #AITrends #NextGenAI #ScalableAI #AIApplications #FutureOfAI