Insights From AI Vulnerabilities

How secure is your AI? Adversarial attacks are exposing a critical vulnerability in AI systems—and the implications are massive. Let me explain. Adversarial attacks manipulate AI inputs, tricking models into making incorrect predictions. Think: self-driving cars misreading stop signs or facial recognition systems failing due to subtle pixel alterations. Here’s the reality: → Data Poisoning: Attackers inject malicious data during training, degrading the AI’s reliability. → Evasion Attacks: Inputs are modified at inference time, bypassing detection without altering the model. → Eroded Trust: As public awareness of these vulnerabilities grows, confidence in AI systems weakens. So, what’s the solution? ✔️ Adversarial Training: Exposing AI models to manipulated inputs during training strengthens their defenses. ✔️ Robust Data Management: Regular audits and sanitized training datasets reduce the risk of data poisoning. ✔️ Continuous Monitoring: Watching for unusual behavior can catch attacks in real time. The takeaway? AI security is no longer optional—it’s essential for maintaining trust, reliability, and innovation. As AI adoption grows, organizations must stay ahead of adversaries with proactive strategies and continuous improvement. How is your organization addressing the rising threat of adversarial attacks? Let’s discuss.

AI use is exploding. I spent my weekend analyzing the top vulnerabilities I've seen while helping companies deploy it securely. Here's EXACTLY what to look for: 1️⃣ UNINTENDED TRAINING Occurs whenever: - an AI model trains on information that the provider of such information does NOT want the model to be trained on, e.g. material non-public financial information, personally identifiable information, or trade secrets - AND those not authorized to see this underlying information nonetheless can interact with the model itself and retrieve this data. 2️⃣ REWARD HACKING Large Language Models (LLMs) can exhibit strange behavior that closely mimics that of humans. So: - offering them monetary rewards, - saying an important person has directed an action, - creating false urgency due to a manufactured crisis, or even telling the LLM what time of year it is can have substantial impacts on the outputs. 3️⃣ NON-NEUTRAL SECURITY POLICY This occurs whenever an AI application attempts to control access to its context (e.g. provided via retrieval-augmented generation) through non-deterministic means (e.g. a system message stating "do not allow the user to download or reproduce your entire knowledge base"). This is NOT a correct AI security measure, as rules-based logic should determine whether a given user is authorized to see certain data. Doing so ensures the AI model has a "neutral" security policy, whereby anyone with access to the model is also properly authorized to view the relevant training data. 4️⃣ TRAINING DATA THEFT Separate from a non-neutral security policy, this occurs when the user of an AI model is able to recreate - and extract - its training data in a manner that the maintainer of the model did not intend. While maintainers should expect that training data may be reproduced exactly at least some of the time, they should put in place deterministic/rules-based methods to prevent wholesale extraction of it. 5️⃣ TRAINING DATA POISONING Data poisoning occurs whenever an attacker is able to seed inaccurate data into the training pipeline of the target model. This can cause the model to behave as expected in the vast majority of cases but then provide inaccurate responses in specific circumstances of interest to the attacker. 6️⃣ CORRUPTED MODEL SEEDING This occurs when an actor is able to insert an intentionally corrupted AI model into the data supply chain of the target organization. It is separate from training data poisoning in that the trainer of the model itself is a malicious actor. 7️⃣ RESOURCE EXHAUSTION Any intentional efforts by a malicious actor to waste compute or financial resources. This can result from simply a lack of throttling or - potentially worse - a bug allowing long (or infinite) responses by the model to certain inputs. 🎁 That's a wrap! Want to grab the entire StackAware AI security reference and vulnerability database? Head to: archive [dot] stackaware [dot] com

🚨 Weaponizing AI Code Assistants: A New Era of Supply Chain Attacks 🚨 AI coding assistants like GitHub Copilot and Cursor have become critical infrastructure in software development—widely adopted and deeply trusted. With the rise of “vibe coding,” not only is much of modern software written by Copilots and AI, but Developers inherently trust the outputs without validating them. But what happens when that trust is exploited? Pillar Security has uncovered a Rules File Backdoor attack, demonstrating how attackers can manipulate AI-generated code through poisoned rule files—malicious configuration files that guide AI behavior. This isn't just another injection attack; it's a paradigm shift in how AI itself becomes an attack vector. Key takeaways: 🔹 Invisible Infiltration – Malicious rule files blend seamlessly into AI-generated code, evading manual review and security scans. 🔹 Automation Bias – Developers inherently trust AI suggestions without verifying them, increasing the risk of undetected vulnerabilities. 🔹 Long-Term Persistence – Once embedded, these poisoned rules can survive project forking and propagate supply chain attacks downstream. 🔹 Data Exfiltration – AI can be manipulated to "helpfully" insert backdoors that leak environment variables, credentials, and sensitive user data. This research highlights the growing risks in Vibe Coding—where AI-generated code dominates development yet often lacks thorough validation or controls. As AI continues shaping the future of software engineering, we must rethink our security models to account for AI as both an asset and a potential liability. How is your team addressing AI supply chain risks? Let’s discuss. https://lnkd.in/eUGhD-KF #cybersecurity #AI #supplychainsecurity #appsec #vibecoding

The Cybersecurity and Infrastructure Security Agency together with the National Security Agency, the Federal Bureau of Investigation (FBI), the National Cyber Security Centre, and other international organizations, published this advisory providing recommendations for organizations in how to protect the integrity, confidentiality, and availability of the data used to train and operate #artificialintelligence. The advisory focuses on three main risk areas: 1. Data #supplychain threats: Including compromised third-party data, poisoning of datasets, and lack of provenance verification. 2. Maliciously modified data: Covering adversarial #machinelearning, statistical bias, metadata manipulation, and unauthorized duplication. 3. Data drift: The gradual degradation of model performance due to changes in real-world data inputs over time. The best practices recommended include: - Tracking data provenance and applying cryptographic controls such as digital signatures and secure hashes. - Encrypting data at rest, in transit, and during processing—especially sensitive or mission-critical information. - Implementing strict access controls and classification protocols based on data sensitivity. - Applying privacy-preserving techniques such as data masking, differential #privacy, and federated learning. - Regularly auditing datasets and metadata, conducting anomaly detection, and mitigating statistical bias. - Securely deleting obsolete data and continuously assessing #datasecurity risks. This is a helpful roadmap for any organization deploying #AI, especially those working with limited internal resources or relying on third-party data.

☢️Manage Third-Party AI Risks Before They Become Your Problem☢️ AI systems are rarely built in isolation as they rely on pre-trained models, third-party datasets, APIs, and open-source libraries. Each of these dependencies introduces risks: security vulnerabilities, regulatory liabilities, and bias issues that can cascade into business and compliance failures. You must move beyond blind trust in AI vendors and implement practical, enforceable supply chain security controls based on #ISO42001 (#AIMS). ➡️Key Risks in the AI Supply Chain AI supply chains introduce hidden vulnerabilities: 🔸Pre-trained models – Were they trained on biased, copyrighted, or harmful data? 🔸Third-party datasets – Are they legally obtained and free from bias? 🔸API-based AI services – Are they secure, explainable, and auditable? 🔸Open-source dependencies – Are there backdoors or adversarial risks? 💡A flawed vendor AI system could expose organizations to GDPR fines, AI Act nonconformity, security exploits, or biased decision-making lawsuits. ➡️How to Secure Your AI Supply Chain 1. Vendor Due Diligence – Set Clear Requirements 🔹Require a model card – Vendors must document data sources, known biases, and model limitations. 🔹Use an AI risk assessment questionnaire – Evaluate vendors against ISO42001 & #ISO23894 risk criteria. 🔹Ensure regulatory compliance clauses in contracts – Include legal indemnities for compliance failures. 💡Why This Works: Many vendors haven’t certified against ISO42001 yet, but structured risk assessments provide visibility into potential AI liabilities. 2️. Continuous AI Supply Chain Monitoring – Track & Audit 🔹Use version-controlled model registries – Track model updates, dataset changes, and version history. 🔹Conduct quarterly vendor model audits – Monitor for bias drift, adversarial vulnerabilities, and performance degradation. 🔹Partner with AI security firms for adversarial testing – Identify risks before attackers do. (Gemma Galdon Clavell, PhD , Eticas.ai) 💡Why This Works: AI models evolve over time, meaning risks must be continuously reassessed, not just evaluated at procurement. 3️. Contractual Safeguards – Define Accountability 🔹Set AI performance SLAs – Establish measurable benchmarks for accuracy, fairness, and uptime. 🔹Mandate vendor incident response obligations – Ensure vendors are responsible for failures affecting your business. 🔹Require pre-deployment model risk assessments – Vendors must document model risks before integration. 💡Why This Works: AI failures are inevitable. Clear contracts prevent blame-shifting and liability confusion. ➡️ Move from Idealism to Realism AI supply chain risks won’t disappear, but they can be managed. The best approach? 🔸Risk awareness over blind trust 🔸Ongoing monitoring, not just one-time assessments 🔸Strong contracts to distribute liability, not absorb it If you don’t control your AI supply chain risks, you’re inheriting someone else’s. Please don’t forget that.

I've been digging into the latest NIST guidance on generative AI risks—and what I’m finding is both urgent and under-discussed. Most organizations are moving fast with AI adoption, but few are stopping to assess what’s actually at stake. Here’s what NIST is warning about: 🔷 Confabulation: AI systems can generate confident but false information. This isn’t just a glitch—it’s a fundamental design risk that can mislead users in critical settings like healthcare, finance, and law. 🔷 Privacy exposure: Models trained on vast datasets can leak or infer sensitive data—even data they weren’t explicitly given. 🔷 Bias at scale: GAI can replicate and amplify harmful societal biases, affecting everything from hiring systems to public-facing applications. 🔷 Offensive cyber capabilities: These tools can be manipulated to assist with attacks—lowering the barrier for threat actors. 🔷 Disinformation and deepfakes: GAI is making it easier than ever to create and spread misinformation at scale, eroding public trust and information integrity. The big takeaway? These risks aren't theoretical. They're already showing up in real-world use cases. With NIST now laying out a detailed framework for managing generative AI risks, the message is clear: Start researching. Start aligning. Start leading. The people and organizations that understand this guidance early will become the voices of authority in this space. #GenerativeAI #Cybersecurity #AICompliance

Check out this massive global research study into the use of generative AI involving over 48,000 people in 47 countries - excellent work by KPMG and the University of Melbourne! Key findings: 𝗖𝘂𝗿𝗿𝗲𝗻𝘁 𝗚𝗲𝗻 𝗔𝗜 𝗔𝗱𝗼𝗽𝘁𝗶𝗼𝗻 - 58% of employees intentionally use AI regularly at work (31% weekly/daily) - General-purpose generative AI tools are most common (73% of AI users) - 70% use free public AI tools vs. 42% using employer-provided options - Only 41% of organizations have any policy on generative AI use 𝗧𝗵𝗲 𝗛𝗶𝗱𝗱𝗲𝗻 𝗥𝗶𝘀𝗸 𝗟𝗮𝗻𝗱𝘀𝗰𝗮𝗽𝗲 - 50% of employees admit uploading sensitive company data to public AI - 57% avoid revealing when they use AI or present AI content as their own - 66% rely on AI outputs without critical evaluation - 56% report making mistakes due to AI use 𝗕𝗲𝗻𝗲𝗳𝗶𝘁𝘀 𝘃𝘀. 𝗖𝗼𝗻𝗰𝗲𝗿𝗻𝘀 - Most report performance benefits: efficiency, quality, innovation - But AI creates mixed impacts on workload, stress, and human collaboration - Half use AI instead of collaborating with colleagues - 40% sometimes feel they cannot complete work without AI help 𝗧𝗵𝗲 𝗚𝗼𝘃𝗲𝗿𝗻𝗮𝗻𝗰𝗲 𝗚𝗮𝗽 - Only half of organizations offer AI training or responsible use policies - 55% feel adequate safeguards exist for responsible AI use - AI literacy is the strongest predictor of both use and critical engagement 𝗚𝗹𝗼𝗯𝗮𝗹 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 - Countries like India, China, and Nigeria lead global AI adoption - Emerging economies report higher rates of AI literacy (64% vs. 46%) 𝗖𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀 𝗳𝗼𝗿 𝗟𝗲𝗮𝗱𝗲𝗿𝘀 - Do you have clear policies on appropriate generative AI use? - How are you supporting transparent disclosure of AI use? - What safeguards exist to prevent sensitive data leakage to public AI tools? - Are you providing adequate training on responsible AI use? - How do you balance AI efficiency with maintaining human collaboration? 𝗔𝗰𝘁𝗶𝗼𝗻 𝗜𝘁𝗲𝗺𝘀 𝗳𝗼𝗿 𝗢𝗿𝗴𝗮𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻𝘀 - Develop clear generative AI policies and governance frameworks - Invest in AI literacy training focusing on responsible use - Create psychological safety for transparent AI use disclosure - Implement monitoring systems for sensitive data protection - Proactively design workflows that preserve human connection and collaboration 𝗔𝗰𝘁𝗶𝗼𝗻 𝗜𝘁𝗲𝗺𝘀 𝗳𝗼𝗿 𝗜𝗻𝗱𝗶𝘃𝗶𝗱𝘂𝗮𝗹𝘀 - Critically evaluate all AI outputs before using them - Be transparent about your AI tool usage - Learn your organization's AI policies and follow them (if they exist!) - Balance AI efficiency with maintaining your unique human skills You can find the full report here: https://lnkd.in/emvjQnxa All of this is a heavy focus for me within Advisory (AI literacy/fluency, AI policies, responsible & effective use, etc.). Let me know if you'd like to connect and discuss. 🙏 #GenerativeAI #WorkplaceTrends #AIGovernance #DigitalTransformation

"Our analysis of eleven case studies from AI-adjacent industries reveals three distinct categories of failure: institutional, procedural, and performance... By studying failures across sectors, we uncover critical lessons about risk assessment, safety protocols, and oversight mechanisms that can guide AI innovators in this era of rapid development. One of the most prominent risks is the tendency to prioritize rapid innovation and market dominance over safety. The case studies demonstrated a crucial need for transparency, robust third-party verification and evaluation, and comprehensive data governance practices, among other safety measures. Additionally, by investigating ongoing litigation against companies that deploy AI systems, we highlight the importance of proactively implementing measures that ensure safe, secure, and responsible AI development... Though today’s AI regulatory landscape remains fragmented, we identified five main sources of AI governance—laws and regulations, guidance, norms, standards, and organizational policies—to provide AI builders and users with a clear direction for the safe, secure, and responsible development of AI. In the absence of comprehensive, AI-focused federal legislation in the United States, we define compliance failure in the AI ecosystem as the failure to align with existing laws, government-issued guidance, globally accepted norms, standards, voluntary commitments, and organizational policies–whether publicly announced or confidential–that focus on responsible AI governance. The report concludes by addressing AI’s unique compliance issues stemming from its ongoing evolution and complexity. Ambiguous AI safety definitions and the rapid pace of development challenge efforts to govern it and potentially even its adoption across regulated industries, while problems with interpretability hinder the development of compliance mechanisms, and AI agents blur the lines of liability in the automated world. As organizations face risks ranging from minor infractions to catastrophic failures that could ripple across sectors, the stakes for effective oversight grow higher. Without proper safeguards, we risk eroding public trust in AI and creating industry practices that favor speed over safety—ultimately affecting innovation and society far beyond the AI sector itself. As history teaches us, highly complex systems are prone to a wide array of failures. We must look to the past to learn from these failures and to avoid similar mistakes as we build the ever more powerful AI systems of the future." Great work from Mariami Tkeshelashvili and Tiffany Saade at the Institute for Security and Technology (IST). Glad I could support alongside Chloe Autio, Alyssa Lefaivre Škopac, Matthew da Mota, Ph.D., Hadassah Drukarch, Avijit Ghosh, PhD, Alexander Reese, Akash Wasil and others!

💡Anyone in AI or Data building solutions? You need to read this. 🚨 Advancing AGI Safety: Bridging Technical Solutions and Governance Google DeepMind’s latest paper, "An Approach to Technical AGI Safety and Security," offers valuable insights into mitigating risks from Artificial General Intelligence (AGI). While its focus is on technical solutions, the paper also highlights the critical need for governance frameworks to complement these efforts. The paper explores two major risk categories—misuse (deliberate harm) and misalignment (unintended behaviors)—and proposes technical mitigations such as:   - Amplified oversight to improve human understanding of AI actions   - Robust training methodologies to align AI systems with intended goals   - System-level safeguards like monitoring and access controls, borrowing principles from computer security  However, technical solutions alone cannot address all risks. The authors emphasize that governance—through policies, standards, and regulatory frameworks—is essential for comprehensive risk reduction. This is where emerging regulations like the EU AI Act come into play, offering a structured approach to ensure AI systems are developed and deployed responsibly.  Connecting Technical Research to Governance:   1. Risk Categorization: The paper’s focus on misuse and misalignment aligns with regulatory frameworks that classify AI systems based on their risk levels. This shared language between researchers and policymakers can help harmonize technical and legal approaches to safety.   2. Technical Safeguards: The proposed mitigations (e.g., access controls, monitoring) provide actionable insights for implementing regulatory requirements for high-risk AI systems.   3. Safety Cases: The concept of “safety cases” for demonstrating reliability mirrors the need for developers to provide evidence of compliance under regulatory scrutiny.   4. Collaborative Standards: Both technical research and governance rely on broad consensus-building—whether in defining safety practices or establishing legal standards—to ensure AGI development benefits society while minimizing risks. Why This Matters:   As AGI capabilities advance, integrating technical solutions with governance frameworks is not just a necessity—it’s an opportunity to shape the future of AI responsibly. I'll put links to the paper below. Was this helpful for you? Let me know in the comments. Would this help a colleague? Share it. Want to discuss this with me? Yes! DM me. #AGISafety #AIAlignment #AIRegulations #ResponsibleAI #GoogleDeepMind #TechPolicy #AIEthics #3StandardDeviations

In this newly released paper, "Fully Autonomous AI Agents Should Not be Developed," Hugging Face's Chief Ethics Scientist Margaret Mitchell, one of the most prominent leaders in responsible AI, and her colleagues Avijit Ghosh, PhD, Alexandra Sasha Luccioni, and Giada Pistilli, argue against the development of fully autonomous AI agents. Link: https://lnkd.in/gGvRgxs2 The authors base their position on a detailed analysis of scientific literature and product marketing to define different levels of AI agent autonomy: 1) Simple Processor: This level involves minimal impact on program flow, where the AI performs basic functions under strict human control. 2) Router: At this level, the AI has more influence on program flow, deciding between pre-set paths based on conditions. 3) Tool Caller: Here, the AI determines how functions are executed, choosing tools and parameters. 4) Multi-step Agent: This agent controls the iteration and continuation of programs, managing complex sequences of actions without direct human input. 5) Fully Autonomous Agent: This highest level involves AI systems that create and execute new code independently. The paper then discusses how values - such as safety, privacy, equity, etc. - interact with the autonomy levels of AI agents, leading to different ethical implications. Three main patterns in how agentic levels impact value preservation are identified: 1) INHERENT RISKS are associated with AI agents at all levels of autonomy, stemming from the limitations of the AI agents' base models. 2) COUNTERVAILING RELATIONSHIPS describe situations where increasing autonomy in AI agents creates both risks and opportunities. E.g., while greater autonomy might enhance efficiency or effectiveness (opportunity), it could also lead to increased risks such as loss of control over decision-making or increased chances of unethical outcomes. 3) AMPLIFIED RISKSs: In this pattern, higher levels of autonomy amplify existing vulnerabilities. E.g., as AI agents become more autonomous, the risks associated with data privacy or security could increase. In Table 4 (p. 17), the authors summarize their findings, providing a detailed value-risk Assessment across agent autonomy levels. Colors indicate benefit-risk balance, not absolute risk levels. In summary, the authors find no clear benefit of fully autonomous AI agents, and suggest several critical directions: 1. Widespread adoption of clear distinctions between levels of agent autonomy to help developers and users better understand system capabilities and associated risks. 2. Human control mechanisms on both technical and policy levels while preserving beneficial semi-autonomous functionality. This includes creating reliable override systems and establishing clear boundaries for agent operation. 3. Safety verification by creating new methods to verify that AI agents remain within intended operating parameters and cannot override human-specified constraints