Understanding Risks of AI Model Usage

"this toolkit shows you how to identify, monitor and mitigate the ‘hidden’ behavioural and organisational risks associated with AI roll-outs. These are the unintended consequences that can arise from how well-intentioned people, teams and organisations interact with AI solutions. Who is this toolkit for? This toolkit is designed for individuals and teams responsible for implementing AI tools and services within organisations and those involved in AI governance. It is intended to be used once you have identified a clear business need for an AI tool and want to ensure that your tool is set up for success. If an AI solution has already been implemented within your organisation, you can use this toolkit to assess risks posed and design a holistic risk management approach. You can use the Mitigating Hidden AI Risks Toolkit to: • Assess the barriers your target users and organisation may experience to using your tool safely and responsibly • Pre-empt the behavioural and organisational risks that could emerge from scaling your AI tools • Develop robust risk management approaches and mitigation strategies to support users, teams and organisations to use your tool safely and responsibly • Design effective AI safety training programmes for your users • Monitor and evaluate the effectiveness of your risk mitigations to ensure you not only minimise risk, but maximise the positive impact of your tool for your organisation" A very practical guide to behavioural considerations in managing risk by Dr Moira Nicolson and others at the UK Cabinet Office, which builds on the MIT AI Risk Repository.

One of the most important contributions of Google DeepMind's new AGI Safety and Security paper is a clean, actionable framing of risk types. Instead of lumping all AI risks into one “doomer” narrative, they break it down into 4 clear categories- with very different implications for mitigation: 1. Misuse → The user is the adversary This isn’t the model behaving badly on its own. It’s humans intentionally instructing it to cause harm- think jailbreak prompts, bioengineering recipes, or social engineering scripts. If we don’t build strong guardrails around access, it doesn’t matter how aligned your model is. Safety = security + control 2. Misalignment → The AI is the adversary The model understands the developer’s intent- but still chooses a path that’s misaligned. It optimizes the reward signal, not the goal behind it. This is the classic “paperclip maximizer” problem, but much more subtle in practice. Alignment isn’t a static checkbox. We need continuous oversight, better interpretability, and ways to build confidence that a system is truly doing what we intend- even as it grows more capable. 3. Mistakes → The world is the adversary Sometimes the AI just… gets it wrong. Not because it’s malicious, but because it lacks the context, or generalizes poorly. This is where brittleness shows up- especially in real-world domains like healthcare, education, or policy. Don’t just test your model- stress test it. Mistakes come from gaps in our data, assumptions, and feedback loops. It's important to build with humility and audit aggressively. 4. Structural Risks → The system is the adversary These are emergent harms- misinformation ecosystems, feedback loops, market failures- that don’t come from one bad actor or one bad model, but from the way everything interacts. These are the hardest problems- and the most underfunded. We need researchers, policymakers, and industry working together to design incentive-aligned ecosystems for AI. The brilliance of this framework: It gives us language to ask better questions. Not just “is this AI safe?” But: - Safe from whom? - In what context? - Over what time horizon? We don’t need to agree on timelines for AGI to agree that risk literacy like this is step one. I’ll be sharing more breakdowns from the paper soon- this is one of the most pragmatic blueprints I’ve seen so far. 🔗Link to the paper in comments. -------- If you found this insightful, do share it with your network ♻️ Follow me (Aishwarya Srinivasan) for more AI news, insights, and educational content to keep you informed in this hyperfast AI landscape 💙

The Cybersecurity and Infrastructure Security Agency together with the National Security Agency, the Federal Bureau of Investigation (FBI), the National Cyber Security Centre, and other international organizations, published this advisory providing recommendations for organizations in how to protect the integrity, confidentiality, and availability of the data used to train and operate #artificialintelligence. The advisory focuses on three main risk areas: 1. Data #supplychain threats: Including compromised third-party data, poisoning of datasets, and lack of provenance verification. 2. Maliciously modified data: Covering adversarial #machinelearning, statistical bias, metadata manipulation, and unauthorized duplication. 3. Data drift: The gradual degradation of model performance due to changes in real-world data inputs over time. The best practices recommended include: - Tracking data provenance and applying cryptographic controls such as digital signatures and secure hashes. - Encrypting data at rest, in transit, and during processing—especially sensitive or mission-critical information. - Implementing strict access controls and classification protocols based on data sensitivity. - Applying privacy-preserving techniques such as data masking, differential #privacy, and federated learning. - Regularly auditing datasets and metadata, conducting anomaly detection, and mitigating statistical bias. - Securely deleting obsolete data and continuously assessing #datasecurity risks. This is a helpful roadmap for any organization deploying #AI, especially those working with limited internal resources or relying on third-party data.

Generative AI: A Powerful Tool, But One That Needs Responsible Use Generative AI is revolutionizing various fields, from creating stunning artwork to crafting compelling marketing copy. But with this power comes responsibility. Here's a look at some critical risks associated with Generative AI and how we can manage them: Risks of Generative AI: Bias and Discrimination: AI models trained on biased data can perpetuate those biases in their outputs. This can lead to discriminatory content or unfair treatment of certain groups. Misinformation and Deepfakes: Generative AI can create highly realistic fake content, like news articles or videos, that cannot be easily distinguished from reality. This poses a severe threat to trust in information. Privacy Concerns: Generative AI models can generate synthetic data that could be used to identify or track individuals without their consent. Job Displacement: As generative AI automates tasks currently done by humans, job displacement is a concern. We need to focus on reskilling and upskilling the workforce. Mitigating the Risks: Data Quality and Fairness: Ensure training data is diverse, representative, and free from bias. Develop fairness metrics to monitor and mitigate bias in AI outputs. Transparency and Explainability: Develop transparent AI models in their decision-making processes. This allows users to understand how the AI arrived at a particular output and identify potential biases. Regulation and Governance: Establish clear guidelines and regulations for developing and deploying Generative AI to ensure responsible use. Education and Awareness: Educate the public about the capabilities and limitations of Generative AI. This helps people critically evaluate AI-generated content and identify potential risks. #generativeai #artificialintelligence #riskmanagement

I've been digging into the latest NIST guidance on generative AI risks—and what I’m finding is both urgent and under-discussed. Most organizations are moving fast with AI adoption, but few are stopping to assess what’s actually at stake. Here’s what NIST is warning about: 🔷 Confabulation: AI systems can generate confident but false information. This isn’t just a glitch—it’s a fundamental design risk that can mislead users in critical settings like healthcare, finance, and law. 🔷 Privacy exposure: Models trained on vast datasets can leak or infer sensitive data—even data they weren’t explicitly given. 🔷 Bias at scale: GAI can replicate and amplify harmful societal biases, affecting everything from hiring systems to public-facing applications. 🔷 Offensive cyber capabilities: These tools can be manipulated to assist with attacks—lowering the barrier for threat actors. 🔷 Disinformation and deepfakes: GAI is making it easier than ever to create and spread misinformation at scale, eroding public trust and information integrity. The big takeaway? These risks aren't theoretical. They're already showing up in real-world use cases. With NIST now laying out a detailed framework for managing generative AI risks, the message is clear: Start researching. Start aligning. Start leading. The people and organizations that understand this guidance early will become the voices of authority in this space. #GenerativeAI #Cybersecurity #AICompliance

Insightful Sunday read regarding AI governance and risk. This framework brings some much-needed structure to AI governance in national security, especially in sensitive areas like privacy, rights, and high-stakes decision-making. The sections on restricted uses of AI make it clear that AI should not replace human judgment, particularly in scenarios impacting civil liberties or public trust. This is particularly relevant for national security contexts where public trust is essential, yet easily eroded by perceived overreach or misuse. The emphasis on impact assessments and human oversight is both pragmatic and proactive. AI is powerful, but without proper guardrails, it’s easy for its application to stray into gray areas, particularly in national security. The framework’s call for thorough risk assessments, documented benefits, and mitigated risks is forward-thinking, aiming to balance AI’s utility with caution. Another strong point is the training requirement. AI can be a black box for many users, so the framework rightly mandates that users understand both the tools’ potential and limitations. This also aligns well with the rising concerns around “automation bias,” where users might overtrust AI simply because it’s “smart.” The creation of an oversight structure through CAIOs and Governance Boards shows a commitment to transparency and accountability. It might even serve as a model for non-security government agencies as they adopt AI, reinforcing responsible and ethical AI usage across the board. Key Points: AI Use Restrictions: Strict limits on certain AI applications, particularly those that could infringe on civil rights, civil liberties, or privacy. Specific prohibitions include tracking individuals based on protected rights, inferring sensitive personal attributes (e.g., religion, gender identity) from biometrics, and making high-stakes decisions like immigration status solely based on AI. High-Impact AI and Risk Management: AI that influences major decisions, particularly in national security and defense, must undergo rigorous testing, oversight, and impact assessment. Cataloguing and Monitoring: A yearly inventory of high-impact AI applications, including data on their purpose, benefits, and risks, is required. This step is about creating a transparent and accountable record of AI use, aimed at keeping all deployed systems in check and manageable. Training and Accountability: Agencies are tasked with ensuring personnel are trained to understand the AI tools they use, especially those in roles with significant decision-making power. Training focuses on preventing overreliance on AI, addressing biases, and understanding AI’s limitations. Oversight Structure: A Chief AI Officer (CAIO) is essential within each agency to oversee AI governance and promote responsible AI use. An AI Governance Board is also mandated to oversee all high-impact AI activities within each agency, keeping them aligned with the framework’s principles.

I'm increasingly convinced that we need to treat "AI privacy" as a distinct field within privacy, separate from but closely related to "data privacy". Just as the digital age required the evolution of data protection laws, AI introduces new risks that challenge existing frameworks, forcing us to rethink how personal data is ingested and embedded into AI systems. Key issues include: 🔹 Mass-scale ingestion – AI models are often trained on huge datasets scraped from online sources, including publicly available and proprietary information, without individuals' consent. 🔹 Personal data embedding – Unlike traditional databases, AI models compress, encode, and entrench personal data within their training, blurring the lines between the data and the model. 🔹 Data exfiltration & exposure – AI models can inadvertently retain and expose sensitive personal data through overfitting, prompt injection attacks, or adversarial exploits. 🔹 Superinference – AI uncovers hidden patterns and makes powerful predictions about our preferences, behaviours, emotions, and opinions, often revealing insights that we ourselves may not even be aware of. 🔹 AI impersonation – Deepfake and generative AI technologies enable identity fraud, social engineering attacks, and unauthorized use of biometric data. 🔹 Autonomy & control – AI may be used to make or influence critical decisions in domains such as hiring, lending, and healthcare, raising fundamental concerns about autonomy and contestability. 🔹 Bias & fairness – AI can amplify biases present in training data, leading to discriminatory outcomes in areas such as employment, financial services, and law enforcement. To date, privacy discussions have focused on data - how it's collected, used, and stored. But AI challenges this paradigm. Data is no longer static. It is abstracted, transformed, and embedded into models in ways that challenge conventional privacy protections. If "AI privacy" is about more than just the data, should privacy rights extend beyond inputs and outputs to the models themselves? If a model learns from us, should we have rights over it? #AI #AIPrivacy #Dataprivacy #Dataprotection #AIrights #Digitalrights

A lot of companies think they’re “safe” from AI compliance risks simply because they haven’t formally adopted AI. But that’s a dangerous assumption—and it’s already backfiring for some organizations. Here’s what’s really happening— Employees are quietly using ChatGPT, Claude, Gemini, and other tools to summarize customer data, rewrite client emails, or draft policy documents. In some cases, they’re even uploading sensitive files or legal content to get a “better” response. The organization may not have visibility into any of it. This is what’s called Shadow AI—unauthorized or unsanctioned use of AI tools by employees. Now, here’s what a #GRC professional needs to do about it: 1. Start with Discovery: Use internal surveys, browser activity logs (if available), or device-level monitoring to identify which teams are already using AI tools and for what purposes. No blame—just visibility. 2. Risk Categorization: Document the type of data being processed and match it to its sensitivity. Are they uploading PII? Legal content? Proprietary product info? If so, flag it. 3. Policy Design or Update: Draft an internal AI Use Policy. It doesn’t need to ban tools outright—but it should define: • What tools are approved • What types of data are prohibited • What employees need to do to request new tools 4. Communicate and Train: Employees need to understand not just what they can’t do, but why. Use plain examples to show how uploading files to a public AI model could violate privacy law, leak IP, or introduce bias into decisions. 5. Monitor and Adjust: Once you’ve rolled out your first version of the policy, revisit it every 60–90 days. This field is moving fast—and so should your governance. This can happen anywhere: in education, real estate, logistics, fintech, or nonprofits. You don’t need a team of AI engineers to start building good governance. You just need visibility, structure, and accountability. Let’s stop thinking of AI risk as something “only tech companies” deal with. Shadow AI is already in your workplace—you just haven’t looked yet.

I was at Hugging Face during the critical year before and after ChatGPT's release. One thing became painfully clear: the ways AI systems can fail are exponentially more numerous than traditional software. Enterprise leaders today are under-estimating AI risks. Data privacy and hallucinations are just the tip of the iceberg. What enterprises aren't seeing: The gap between perceived and actual AI failure modes is staggering. - Enterprises think they're facing 10 potential failure scenarios…  - when the reality is closer to 100. AI risks fall into two distinct categories that require completely different approaches: Internal risks: When employees use AI tools like ChatGPT, they often inadvertently upload proprietary information. Your company's competitive edge is now potentially training competitor's models. Despite disclaimer pop-ups, this happens constantly. External risks: These are far more dangerous. When your customers interact with your AI-powered experiences, a single harmful response can destroy brand trust built over decades. Remember when Gemini's image generation missteps wiped billions off Google's market cap? Shout out to Dr. Ratinder, CTO Security and Gen AI, Pure Storage. When I got on a call with Ratinder, he very enthusiastically explained to me their super comprehensive approach: ✅ Full DevSecOps program with threat modeling, code scanning, and pen testing, secure deployment and operations ✅ Security policy generation system that enforces rules on all inputs/outputs ✅ Structured prompt engineering with 20+ techniques ✅ Formal prompt and model evaluation framework ✅ Complete logging via Splunk for traceability ✅ Third-party pen testing certification for customer trust center ✅ OWASP Top 10 framework compliance ✅ Tests for jailbreaking attempts during the development phase Their rigor is top-class… a requirement for enterprise-grade AI. For most companies, external-facing AI requires 2-3x the guardrails of internal systems. Your brand reputation simply can't afford the alternative. Ask yourself: What AI risk factors is your organization overlooking? The most dangerous ones are likely those you haven't even considered.

Responsible data development is at the core of Responsible AI (RAI). If a training dataset was created poorly (under-represented, skewed data) this will lead to a biased model. In AI development, using real data has privacy, ethical, and IP implications, to name a few. On the other hand, using synthetic (AI-generated) data is not a panacea (as much as it’s been hailed). It leads to other kinds of downstream issues that need to be taken into account. This paper explores two key risks of using synthetic data in AI model development: 1.      Diversity-washing (synthetic data can give the appearance of diversity) 2.      Consent circumvention (consent stops being a “procedural hook” that limits downstream harms from AI model use and this – along with data source obfuscation - complicates enforcement) The paper focuses on facial recognition technology (FRT) highlighting the risks of using synthetic data, and the trade-offs between utility, fidelity, and privacy. It’s important to develop participatory governance models along with data lineage and transparency which are crucial when it comes to mitigating these risks.