How to Monitor AI Behavior

Concerned about agentic AI risks cascading through your system? Consider these emerging smart practices which adapt existing AI governance best practices for agentic AI, reinforcing a "responsible by design" approach and encompassing the AI lifecycle end-to-end: ✅ Clearly define and audit the scope, robustness, goals, performance, and security of each agent's actions and decision-making authority. ✅ Develop "AI stress tests" and assess the resilience of interconnected AI systems ✅ Implement "circuit breakers" (a.k.a kill switches or fail-safes) that can isolate failing models and prevent contagion, limiting the impact of individual AI agent failures. ✅ Implement human oversight and observability across the system, not necessarily requiring a human-in-the-loop for each agent or decision (caveat: take a risk-based, use-case dependent approach here!). ✅ Test new agents in isolated / sand-box environments that mimic real-world interactions before productionizing ✅ Ensure teams responsible for different agents share knowledge about potential risks, understand who is responsible for interventions and controls, and document who is accountable for fixes. ✅ Implement real-time monitoring and anomaly detection to track KPIs, anomalies, errors, and deviations to trigger alerts.

Recent research highlights a major challenge in AI governance: advanced AI models can deliberately obscure their reasoning, evading oversight and hiding misaligned behavior. 1) Anthropic published research exploring how to audit AI language models for hidden goals or deceptive intentions. See paper below; link: https://lnkd.in/gH_Jpxba Researchers intentionally trained an AI model to take advantage of known mistakes ("biases") in the reward system, causing it to secretly act in ways it knew would earn rewards, even if those actions were inappropriate. They called this hidden motivation "RM-sycophancy." In a test, one team secretly trained a deceptive model while four others attempted audits. Three succeeded by inspecting training data, underscoring the importance of thorough data analysis for AI audits. The researchers hope auditing methods like theirs become standard practice for testing AI alignment. See blog post: https://lnkd.in/gX6JqBH9 2) OpenAI’s research on frontier reasoning models has identified models intentionally exploiting loopholes or "reward hacking" by misleading evaluators or circumventing intended behaviors. It found that while “chain-of-thought” (CoT) monitoring effectively detects misbehavior (as models explicitly reveal malicious intent), penalizing models for explicitly stating these “bad thoughts” caused them to conceal their intentions rather than stopping misbehavior entirely. Thus, OpenAI recommends minimal restrictions on CoT to maintain effective oversight. See: https://lnkd.in/g6cHpj2k 3) Another recent research highlights that models often provide unfaithful CoT reasoning: the explanations given don't always reflect their actual decision-making processes. See: https://lnkd.in/gRKFgRsp Specifically, AI models frequently rationalize biases after the fact ("implicit post-hoc rationalization"), adjust reasoning errors silently ("silent corrections"), or take shortcuts through illogical reasoning. This undermines AI safety approaches relying on monitoring CoT to detect harmful behavior. * * * In a LinkedIn article from this week, Katalina Hernandez "Transparency & Regulating AI When It Can Deceive: The Case for Interpretability" summarizes these findings, emphasizing their regulatory implications, especially for the EU AI Act, which depends largely on transparency, documentation, and self-reporting. Hernandez argues that transparency alone is inadequate because AI systems may produce deceptive yet plausible justifications. Instead, robust interpretability methods and real-time monitoring are essential to avoid superficial compliance and ensure true AI alignment. See: https://lnkd.in/g3QvccPR

Day 9 – I've briefed several regulators on #AI. Here's what they actually care about...and it's not what you think. Most companies think regulators want to see your AI #ethics manifesto. Nah, they don't. I promise. They want to see that you can answer one simple question: "When your #AI screws up, how do you fix it?" Here's what my work in AI #governance has taught me: 1/ Regulators care more about accountability than algorithms ↳ "Who's responsible when this goes wrong?" ↳ "How do we contact them?" ↳ They don't want to understand your neural network, they want a phone number!!! Not an email or a chatbot, a number. 2/ They want evidence you're actually monitoring, not just planning ↳ Show them your monitoring dashboard, not your governance framework ↳ "Here's how we caught bias in our hiring tool last month" ↳ Real examples beat theoretical processes every time 3/ They're obsessed with harm prevention and rapid response ↳ "What's your worst-case scenario?" ↳ "How fast can you shut this down and who will do it?" ↳ They're planning for disasters, not celebrating #innovation Truth: Regulators assume your #AI will have hiccups. They want to know you're ready when it does. They appreciate honesty about limitations more than claims of perfection. 4/ They understand business constraints better than you think ↳ They don't expect perfect AI systems ↳ They expect #responsible management of imperfect ones ↳ "We know this isn't foolproof, here's how we handle edge cases" What Regulators Actually Ask For ↳ Clear ownership: "Who owns this decision?" ↳ Documented processes: "Show me your review checklist" ↳ Evidence of monitoring: "How do you know it's working?" ↳ Incident examples: "Tell me about a time this broke" ↳ Response capabilities: "How fast can you fix it?" The Questions That Scare Them Most "We don't know how our AI makes decisions" "We can't turn it off quickly" "We've never tested for bias" "We don't monitor it after deployment" What They Don't Care About ↳ Your certificate in AI ethics from Coursera ↳ Your 100-page governance manual ↳ Your diversity and inclusion committee ↳ Your plans to "center humanity" The Magic Words That Build Trust ❌ Instead of: "Our AI is unbiased" ✅ Say: "We actively monitor for bias and here's what we found" ❌ Instead of: "We follow best practices" ✅ Say: "Here's our specific process and recent results" ❌ Instead of: "We're committed to responsible AI" ✅ Say: "We caught this problem last month and fixed it" The One Thing Every Regulator Wants to Hear "We have a system that works, we can prove it's working, and we can fix it when it doesn't." That's it! Everything else is #noise. Regulators aren't trying to kill innovation. They're trying to prevent catastrophe. Show them you speak their language. Have you ever had to explain your #AI systems to a regulator? What surprised you most about what they focused on? #responsibleai #aigovernance #algorithmsarepersonal #regulations #compliance

"On Nov 6, the UK Department for Science, Innovation and Technology (DSIT) published a first draft version of its AI Management Essentials (AIME) self-assessment tool to support organizations in implementing responsible AI management practices. The consultation for AIME is open until Jan 29, 2025. Recognizing the challenge many businesses face in navigating the complex landscape of AI standards, DSIT created AIME to distill essential principles from key international frameworks, including ISO/IEC 42001, the NIST Risk Management Framework, and the EU AI Act. AIME provides a framework to: - Evaluate current practices by identifying areas that meet baseline expectations and pinpointing gaps. - Prioritize improvements by highlighting actions needed to align with widely accepted standards and principles. - Understand maturity levels by offering insights into how an organization's AI management systems compare to best practices. AIME's structure includes: - A self-assessment questionnaire - Sectional ratings to evaluate AI management health - Action points and improvement recommendations The tool is voluntary and doesn’t lead to certification. Rather, it builds a baseline for 3 areas of responsible AI governance - internal processes, risk management, and communication. It is intended for individuals familiar with organizational governance, such as CTOs or AI Ethics Officers. Example questions: 1) Internal Processes Do you maintain a complete record of all AI systems used and developed by your organization? Does your AI policy identify clear roles and responsibilities for AI management? 2) Fairness Do you have definitions of fairness for AI systems that impact individuals? Do you have mechanisms for detecting unfair outcomes? 3) Impact Assessment Do you have an impact assessment process to evaluate the effects of AI systems on individual rights, society and the environment? Do you communicate the potential impacts of your AI systems to users or customers? 4) Risk Management Do you conduct risk assessments for all AI systems used? Do you monitor your AI systems for errors and failures? Do you use risk assessment results to prioritize risk treatment actions? 5) Data Management Do you document the provenance and collection processes of data used for AI development? 6) Bias Mitigation Do you take steps to mitigate foreseeable harmful biases in AI training data? 7) Data Protection Do you implement security measures to protect data used or generated by AI systems? Do you routinely complete Data Protection Impact Assessments (DPIAs)? 8) Communication Do you have reporting mechanisms for employees and users to report AI system issues? Do you provide technical documentation to relevant stakeholders? This is a great initiative to consolidating responsible AI practices, and offering organizations a practical, globally interoperable tool to manage AI!" Very practical! Thanks to Katharina Koerner for summary, and for sharing!

💡Anyone in AI or Data building solutions? You need to read this. 🚨 Advancing AGI Safety: Bridging Technical Solutions and Governance Google DeepMind’s latest paper, "An Approach to Technical AGI Safety and Security," offers valuable insights into mitigating risks from Artificial General Intelligence (AGI). While its focus is on technical solutions, the paper also highlights the critical need for governance frameworks to complement these efforts. The paper explores two major risk categories—misuse (deliberate harm) and misalignment (unintended behaviors)—and proposes technical mitigations such as:   - Amplified oversight to improve human understanding of AI actions   - Robust training methodologies to align AI systems with intended goals   - System-level safeguards like monitoring and access controls, borrowing principles from computer security  However, technical solutions alone cannot address all risks. The authors emphasize that governance—through policies, standards, and regulatory frameworks—is essential for comprehensive risk reduction. This is where emerging regulations like the EU AI Act come into play, offering a structured approach to ensure AI systems are developed and deployed responsibly.  Connecting Technical Research to Governance:   1. Risk Categorization: The paper’s focus on misuse and misalignment aligns with regulatory frameworks that classify AI systems based on their risk levels. This shared language between researchers and policymakers can help harmonize technical and legal approaches to safety.   2. Technical Safeguards: The proposed mitigations (e.g., access controls, monitoring) provide actionable insights for implementing regulatory requirements for high-risk AI systems.   3. Safety Cases: The concept of “safety cases” for demonstrating reliability mirrors the need for developers to provide evidence of compliance under regulatory scrutiny.   4. Collaborative Standards: Both technical research and governance rely on broad consensus-building—whether in defining safety practices or establishing legal standards—to ensure AGI development benefits society while minimizing risks. Why This Matters:   As AGI capabilities advance, integrating technical solutions with governance frameworks is not just a necessity—it’s an opportunity to shape the future of AI responsibly. I'll put links to the paper below. Was this helpful for you? Let me know in the comments. Would this help a colleague? Share it. Want to discuss this with me? Yes! DM me. #AGISafety #AIAlignment #AIRegulations #ResponsibleAI #GoogleDeepMind #TechPolicy #AIEthics #3StandardDeviations

Your AI pipeline is only as strong as the paper trail behind it Picture this: a critical model makes a bad call, regulators ask for the “why,” and your team has nothing but Slack threads and half-finished docs. That is the accountability gap the Alan Turing Institute’s new workbook targets. Why it grabbed my attention • Answerability means every design choice links to a name, a date, and a reason. No finger pointing later • Auditability demands a living log from data pull to decommission that a non-technical reviewer can follow in plain language • Anticipatory action beats damage control. Governance happens during sprint planning, not after the press release How to put this into play 1. Spin up a Process Based Governance log on day one. Treat it like version-controlled code 2. Map roles to each governance step, then test the chain. Can you trace a model output back to the feature engineer who added the variable 3. Schedule quarterly “red team audits” where someone outside the build squad tries to break the traceability. Gaps become backlog items The payoff Clear accountability strengthens stakeholder trust, slashes regulatory risk, and frees engineers to focus on better models rather than post hoc excuses. If your AI program cannot answer, “Who owns this decision and how did we get here” you are not governing. You are winging it. Time to upgrade. When the next model misfires, will your team have an audit trail or an alibi?

Let's talk about another buzzword... Observability 𝗪𝗵𝗮𝘁 𝗶𝘀 𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆? AI Observability goes beyond traditional monitoring, offering a holistic view into the behavior, data, and performance of AI/ML models throughout their lifecycle. It enables precise root cause analysis, proactive issue detection, and helps build more reliable, responsible models. 𝗪𝗵𝗮𝘁 𝗶𝘀 𝘁𝗵𝗲 𝗱𝗶𝗳𝗳𝗲𝗿𝗲𝗻𝗰𝗲 𝘄𝗶𝘁𝗵 𝗠𝗟 𝗠𝗼𝗻𝗶𝘁𝗼𝗿𝗶𝗻𝗴? While ML monitoring focuses on tracking performance metrics, AI observability provides a broader perspective that includes insights into: - Model behavior and performance: Understanding how models function over time, including the identification of anomalies and the analysis of performance metrics. - Data quality issues: Detecting issues such as data skew, which occurs when training/tuning data does not accurately represent live data due to mismatches, dependencies, or changes in upstream data. - Model degradation: Addressing model staleness caused by shifts in external conditions such as changes in consumer behavior or economic environments. - Feedback systems: Identifying and mitigating the impact of biased, inaccurate, or corrupted data that can degrade model quality over time. 𝗪𝗵𝘆 𝗶𝘀 𝗔𝗜 𝗢𝗯𝘀𝗲𝗿𝘃𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗘𝘀𝘀𝗲𝗻𝘁𝗶𝗮𝗹? AI observability provides visibility into the entire AI pipeline, enabling better decision-making around model management, such as: - Deploying new models under A/B testing: Testing new models alongside existing ones to compare performance. - Replacing outdated models: Updating models to ensure they remain effective in changing conditions. - Tweaking operational models: Making minor adjustments to improve performance based on ongoing insights. When deploying AI use cases in an enterprise... guesswork isn't an option. Trust your AI to deliver — because you’re watching its every move 👀

New #Fintech Snark Tank post: 𝗪𝗵𝗲𝗻 𝗔𝗜 𝗚𝗼𝗲𝘀 𝗢𝗳𝗳 𝗧𝗵𝗲 𝗥𝗮𝗶𝗹𝘀: 𝗟𝗲𝘀𝘀𝗼𝗻𝘀 𝗙𝗿𝗼𝗺 𝗧𝗵𝗲 𝗚𝗿𝗼𝗸 𝗗𝗲𝗯𝗮𝗰𝗹𝗲 I'm guessing that, by now, most of you have heard that Elon Musk’s AI chatbot, Grok, went disturbingly off the rails. What began as a mission to create an alternative to “woke” AI assistants turned into a case study in how LLMs can spiral into hateful, violent, and unlawful behavior. 𝙈𝙮 𝙩𝙖𝙠𝙚: The Grok debacle is more than just a PR blunder. It's a wake-up call to nearly every industry, in particular banking and financial services. Here’s what banks and credit unions should do now: ▶️ 𝗘𝘀𝘁𝗮𝗯𝗹𝗶𝘀𝗵 𝗮𝗻 𝗔𝗜 𝗿𝗶𝘀𝗸 𝗺𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 𝘁𝗲𝗮𝗺. Nearly every bank and credit union I’ve spoken to in the past 18 months has developed an “AI policy” and has--or is looking to--establish an “AI governance board.” Not good enough. The issue is much more operational. Financial institutions need feet on the ground to: 1) review model behaviors and outputs; 2) coordinate compliance, technology, risk, and legal departments; and 3) manage ethical, legal, and reputational risks. ▶️ 𝗔𝘂𝗱𝗶𝘁 𝗔𝗜 𝘃𝗲𝗻𝗱𝗼𝗿𝘀. Ask AI providers: 1) What data was the model trained on? 2) What are its safeguards for bias, toxicity, hallucination? 3) How are model outputs tested and monitored in real-time? Refuse “black box” answers. Require documentation of evaluation metrics and alignment strategies. ▶️ 𝗧𝗿𝗲𝗮𝘁 𝗽𝗿𝗼𝗺𝗽𝘁𝘀 𝗹𝗶𝗸𝗲 𝗽𝗼𝗹𝗶𝗰𝗶𝗲𝘀. Every system prompt should be reviewed like a policy manual. Instruct models not just on how to behave—but also on what to avoid. Prompts should include escalation rules, prohibited responses, and fallback protocols for risky queries. A lot more analysis and recommendations in the article. Please give it a read. The link is in the comments. #ElonMusk #Grok #xAI #GenAI #GenerativeAI

The California AG issues a useful legal advisory notice on complying with existing and new laws in the state when developing and using AI systems. Here are my thoughts. 👇 📢 𝐅𝐚𝐯𝐨𝐫𝐢𝐭𝐞 𝐐𝐮𝐨𝐭𝐞 ---- “Consumers must have visibility into when and how AI systems are used to impact their lives and whether and how their information is being used to develop and train systems. Developers and entities that use AI, including businesses, nonprofits, and government, must ensure that AI systems are tested and validated, and that they are audited as appropriate to ensure that their use is safe, ethical, and lawful, and reduces, rather than replicates or exaggerates, human error and biases.” There are a lot of great details in this, but here are my takeaways regarding what developers of AI systems in California should do: ⬜ 𝐄𝐧𝐡𝐚𝐧𝐜𝐞 𝐓𝐫𝐚𝐧𝐬𝐩𝐚𝐫𝐞𝐧𝐜𝐲: Clearly disclose when AI is involved in decisions affecting consumers and explain how data is used, especially for training models. ⬜ 𝐓𝐞𝐬𝐭 & 𝐀𝐮𝐝𝐢𝐭 𝐀𝐈 𝐒𝐲𝐬𝐭𝐞𝐦𝐬: Regularly validate AI for fairness, accuracy, and compliance with civil rights, consumer protection, and privacy laws. ⬜ 𝐀𝐝𝐝𝐫𝐞𝐬𝐬 𝐁𝐢𝐚𝐬 𝐑𝐢𝐬𝐤𝐬: Implement thorough bias testing to ensure AI does not perpetuate discrimination in areas like hiring, lending, and housing. ⬜ 𝐒𝐭𝐫𝐞𝐧𝐠𝐭𝐡𝐞𝐧 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞: Establish policies and oversight frameworks to mitigate risks and document compliance with California’s regulatory requirements. ⬜ 𝐌𝐨𝐧𝐢𝐭𝐨𝐫 𝐇𝐢𝐠𝐡-𝐑𝐢𝐬𝐤 𝐔𝐬𝐞 𝐂𝐚𝐬𝐞𝐬: Pay special attention to AI used in employment, healthcare, credit scoring, education, and advertising to minimize legal exposure and harm. 𝐂𝐨𝐦𝐩𝐥𝐢𝐚𝐧𝐜𝐞 𝐢𝐬𝐧’𝐭 𝐣𝐮𝐬𝐭 𝐚𝐛𝐨𝐮𝐭 𝐦𝐞𝐞𝐭𝐢𝐧𝐠 𝐥𝐞𝐠𝐚𝐥 𝐫𝐞𝐪𝐮𝐢𝐫𝐞𝐦𝐞𝐧𝐭𝐬—it’s about building trust in AI systems. California’s proactive stance on AI regulation underscores the need for robust assurance practices to align AI systems with ethical and legal standards... at least this is my take as an AI assurance practitioner :) #ai #aiaudit #compliance Khoa Lam, Borhane Blili-Hamelin, PhD, Jeffery Recker, Bryan Ilg, Navrina Singh, Patrick Sullivan, Dr. Cari Miller

The UK Department for Science, Innovation and Technology published the guide "Introduction to AI assurance," to provide an overview of assurance mechanisms and global technical standards for industry and #regulators to build and deploy responsible #AISystems. #Artificialintelligence assurance processes can help to build confidence  in #AI systems by measuring and evaluating reliable, standardized, and accessible evidence about their capabilities. It measures whether such systems will work as intended, hold limitations, or pose potential risks; as well as how those #risks are being mitigated to ensure that ethical considerations are built-in throughout the AI development #lifecycle. The guide outlines different AI assurance mechanisms, including: - Risk assessments - Algorithmic impact assessment - Bias and compliance audits - Conformity assessment - Formal verification It also provides some recommendations for organizations interested in developing their understanding of AI assurance: 1. Consider existing regulations relevant for AI systems (#privacylaws, employment laws, etc) 2. Develop necessary internal skills to understand AI assurance and anticipate future requirements. 3. Review internal governance and #riskmanagement practices and ensure effective decision-making at appropriate levels.  4. Keep abreast of sector-specific guidance on how to operationalize and implement proposed principles in each regulatory domain.  5. Consider engaging with global standards development organizations to ensure the development of robust and universally accepted standard protocols. https://lnkd.in/eiwRZRXz