Understanding AI Security Threats

The European Telecommunications Standards Institute (ETSI) AI Threat Ontology provides rigorous foundations for mapping and understanding risks across the AI threat landscape. Key highlights include: • Formalizing AI-specific threat agents, vulnerabilities, and system assets in a structured ontology for adversarial and defensive use. • Modeling AI as both a threat agent and a target with dynamic attributes like observe, learn, and adapt. • Expanding on adversarial goals: from model evasion and training data poisoning to model theft, inversion, and reputational compromise. • Mapping trust relationships across actors: data owners, system builders, training providers, consumers, and outsiders. • Aligning with standards such as OWL, RDF, CVE, CWE, and the CIA (Confidentiality, Integrity, Availability) model. • Accounting for misuse potential across all ML phases: from data curation and transfer learning to sandbox escape and model hallucination. • Addressing cross-domain ontology limitations e.g., how a potato exists in both diet and biology taxonomies. • Emphasizing human-in-the-loop risks such as overtrust in models and attacker-induced alert fatigue. Who should take note: • Security and ontology architects modeling AI-enabled and AI-threat scenarios • Red and blue teams simulating advanced persistent AI threats (APAITs) • Compliance and GRC professionals seeking formal, semantic frameworks for AI assurance. • AI ethics and policy leaders designing layered trust models. Noteworthy aspects: • Built on semantic relationships (subject → predicate → object) to encode AI system risks formally. • Lifecycle-aware guidance: data poisoning → model compromise → deployment misuse. • Defines AI threat agents as extensions of classical agents with real-time learning and behavioral modulation. • Supports both classical and neural-based systems, from expert systems to GANs and DeepFakes. Actionable step: Use the ETSI AI Ontology as a basis to build knowledge graphs and threat modeling frameworks that can observe, reason, and react to adversarial AI risks in real time. Consideration: AI security isn't just about defending models but it is also about defining what security even means when intelligence is both the attacker and the target.

Securing Language Models Enhanced by Generative AI Language models, or LLMs, driven by generative AI, have transformed cybersecurity, yet they pose unique security risks. While they bolster defense mechanisms, they're susceptible to exploitation by adversaries. Let's delve into these risks and the methods attackers use. LLMs in Cybersecurity LLMs bolster cybersecurity defenses but also introduce vulnerabilities. Adversarial AI, for instance, manipulates models to produce desired outcomes. Think of a facial recognition system tricked into accepting a fake face, compromising security. Exploitation Techniques Attackers employ various methods: 1. Evasion Attacks: Crafting deceptive inputs confuses LLMs, leading to misclassifications. For example, adding subtle noise to an image fools an LLM-based security system into seeing harmless content. 2. Boundary Attacks: Exploiting model decision boundaries, attackers manipulate outputs. Techniques like the HopSkipJump Attack subtly alter predictions, like turning a cat into a dog with a single pixel. 3. Transfer-Based Attacks: Attackers train substitute models using data from the target LLM to improve attack success rates. This exploits the transferability of adversarial examples. 4. Model Theft: Stealing LLMs or algorithms enables attackers to identify vulnerabilities or launch more effective attacks, such as spreading misinformation using stolen news-generating models. 5. Data Privacy Attacks: Inadequate privacy measures expose sensitive training data, leading to privacy breaches or inference attacks. 6. Prompt Injection: Injecting malicious prompts into LLMs can lead to incorrect or offensive responses, compromising the model's reliability. For instance, injecting misleading information into customer service chatbots. 7. Sponge Attacks: Denial-of-service attacks overwhelm LLMs with nonsensical queries, disrupting their functionality and preventing legitimate users from accessing the system.

When your AI assistant quietly wipes out your dev environment… and no one notices for 5 days. One more day. One more agent. And this could’ve cascaded into prod — that’s what just happened with Amazon Q. AWS confirmed the details here: https://lnkd.in/gyNb37kS A simple, AI-powered command in the IDE led to automated deletion of hundreds of cloud resources. Not malicious. Not even user error in the traditional sense. Just a quietly dangerous moment where: • Over-privileged AI agents • Poor guardrails • And invisible blast radius combined into a slow-moving disaster. This wasn’t a sophisticated exploit. It was a user asking a natural-language assistant to “delete some stuff.” The AI obliged — with root privileges — and no organizational controls kicked in. No drift detection. No just-in-time escalation. No policy limits. It took 5 days for the team to realize what happened. We’re entering a new phase of AI adoption where the “attack surface” isn’t just about code or infrastructure — it’s about intent interpretation. The interface is the exploit. AI agents need runtime enforcement, scoped credentials, audit memory, and contextual understanding of what should happen — not just what can. Because it’s not just about hallucinations anymore. It’s about irreversible, high-trust actions that feel helpful… until they’re not. If you’re rethinking how to secure AI agents inside your org — from permissions to runtime actions — it’s time to treat them like users, not tools. That’s exactly what we’re doing at Palo Alto Networks. #AI #CyberSecurity #DevSecOps #CloudSecurity #AIGovernance #AIAssistant #SecurityOperations #IdentitySecurity #RuntimeSecurity #GenAI #EnterpriseSecurity #AIagents #LLMsecurity #DataLossPrevention

Is AI Making Cybersecurity More Complex? Credible Voices Discuss Indirect Prompt Hacking The emergence of indirect prompt hacking as a significant cybersecurity concern becomes more significant when discussed by experts like Stewart Baker and Scott Shapiro. Their credentials and focus areas offer a nuanced understanding of the issue, enhancing the discussion. Why Their Perspective Matters - Policy and Legal Insights: Baker's experience in national security policy provides a nuanced understanding of AI vulnerabilities. His skill in data and privacy negotiations enhances the importance of this topic. - Academic Rigor and Real-world Practice: Shapiro combines theoretical and practical aspects of AI vulnerabilities through his academic and directorial roles. Main Points for Consideration - Growing Awareness about Indirect Prompts: Indirect prompts can guide AI models through hidden instructions via external sources like PDFs or URLs. - Exploitation Scenarios: Indirect prompt hacking has varied applications, from altering automatic license plate readers to influencing AI-driven hiring processes. Countermeasures to Consider - Informed Security Controls: Recognizing these vulnerabilities helps in formulating effective security controls. Consider multi-layer controls and external source verification. - Continued Learning: Staff who work with AI should become educated and trained on these vulnerabilities. Awareness often serves as the first step toward improved security practices. Given the caliber of individuals discussing this topic, it's clear that the issue of AI vulnerabilities, especially indirect prompt hacking, merits careful consideration. #cybersecurity #AI #law

𝗧𝗵𝗲 National Institute of Standards and Technology (NIST) 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗔𝗿𝘁𝗶𝗳𝗶𝗰𝗶𝗮𝗹 𝗜𝗻𝘁𝗲𝗹𝗹𝗶𝗴𝗲𝗻𝗰𝗲 𝗣𝗿𝗼𝗳𝗶𝗹𝗲 (𝘁𝗵𝗲 "𝗣𝗿𝗼𝗳𝗶𝗹𝗲") | 𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗼𝗻 𝗶𝘁𝘀 𝗔𝗜 𝗥𝗶𝘀𝗸 𝗠𝗮𝗻𝗮𝗴𝗲𝗺𝗲𝗻𝘁 𝗙𝗿𝗮𝗺𝗲𝘄𝗼𝗿𝗸 (𝗔𝗜 𝗥𝗠𝗙) 𝗳𝗿𝗼𝗺 𝗹𝗮𝘀𝘁 𝘆𝗲𝗮𝗿. This Profile identifies twelve risks associated with Generative AI (GAI), some of which are novel or exacerbated by GAI, including confabulation, toxicity, and homogenization. 🔑 𝗞𝗲𝘆 𝗣𝗼𝗶𝗻𝘁𝘀: 1. 𝗡𝗼𝘃𝗲𝗹 𝗮𝗻𝗱 𝗙𝗮𝗺𝗶𝗹𝗶𝗮𝗿 𝗥𝗶𝘀𝗸𝘀: - Exotic Risks: The Profile introduces risks like confabulation (AI generating false information), toxicity (harmful outputs), and homogenization (lack of diversity in AI outputs). - Cybersecurity Risks: Discovering or lowering barriers for offensive capabilities and expanding the attack surface through novel attack methods. 𝟮. 𝗘𝘅𝗮𝗺𝗽𝗹𝗲𝘀 𝗼𝗳 𝗖𝘆𝗯𝗲𝗿𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗥𝗶𝘀𝗸𝘀: - Large language models identify vulnerabilities in data and writing exploitative code. - GAI-powered co-pilots aiding threat actors in evasion tactics. - Prompt injections can steal data and execute remote code. - Poisoned datasets compromising output integrity. 𝟯. 𝗥𝗲𝗴𝘂𝗹𝗮𝘁𝗼𝗿𝘆 𝗜𝗺𝗽𝗹𝗶𝗰𝗮𝘁𝗶𝗼𝗻𝘀: - Historically, the Federal Trade Commission (FTC) has referred to NIST frameworks in data breach investigations, requiring organizations to adopt measures from the NIST Cybersecurity Framework. - It is likely that NIST's guidance on GAI will similarly be recommended or required in the future. 𝟰. 𝗚𝗔𝗜’𝘀 𝗥𝗼𝗹𝗲 𝗶𝗻 𝗖𝘆𝗯𝗲𝗿𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆: - Despite its risks, GAI also offers benefits for cybersecurity: - Assisting cybersecurity teams and protecting organizations from threats. - Training models to detect weaknesses in applications and code. - Automating vulnerability detection to expedite new code deployment. 𝟱. 𝗣𝗿𝗼𝗮𝗰𝘁𝗶𝘃𝗲 𝗠𝗲𝗮𝘀𝘂𝗿𝗲𝘀: - The Profile offers recommendations to mitigate GAI risks, including: - Refining incident response plans and risk assessments. - Regular adversary testing and tabletop exercises. - Revising contracts to clarify liability and incident handling responsibilities. - Documenting changes throughout the GAI lifecycle, including third-party systems and data storage. 𝟲. 𝗦𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗰 𝗜𝗺𝗽𝗼𝗿𝘁𝗮𝗻𝗰𝗲: - As emphasized by Microsoft's Chief of Security, Charlie Bell, cybersecurity is foundational: “If you don’t solve it, all the other technology stuff just doesn’t happen.” - The AI RMF and the Profile provide guidance on managing GAI risks, crucial for developing secure AI systems. MITRE Center for Internet Security IAPP - International Association of Privacy Professionals ISACA SFIA Foundation ISC2 AICPA The Institute of Internal Auditors Inc. https://lnkd.in/e_Sgwgjr

This week I read through "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations" (NIST.AI.100-2e2023) report. As we navigate the complexities of securing AI systems, understanding the nuances of adversarial attacks becomes critical.   Key Highlights from the Report:   1. **Evasion Attacks:** A reminder that our AI systems are at risk of misclassification attacks. Whether through white-box or black-box methodologies, adversaries are finding sophisticated ways to manipulate testing samples. The report's discussion on mitigations like adversarial training gives us options but also a call to action for enhanced model robustness.   2. **Poisoning Attacks**: The threat of corrupting ML models by tampering with training data is more prevalent than ever. From availability to backdoor poisoning, the strategies employed by attackers are diversifying. The emphasis is on data sanitization and anomaly detection. Vigilance during model training is key.   3. **Privacy Attacks**: The confidentiality of training data and model integrity are under siege. Techniques ranging from data reconstruction to model extraction expose the vulnerabilities in our systems.    4. **Data Loss** (my own, not from the report): When we use Generative AI tools, we need to understand when we upload documents for analysis to these tools, the documents, and the information inside is gone. Traditional DLP systems cannot detect these uploads because the data is unstructured. Organizations will need to update their Acceptable Use policies and Security Awareness programs (at a minimum) to include AI.   As #AI professionals, we must stay informed and proactive in the face of these adversarial threats. Let's take this knowledge and work with our organization on creating AI systems that are not only intelligent but resilient and secure. You can read the report here: https://lnkd.in/e77qqgbM   #ArtificialIntelligence #MachineLearning #AdversarialMachineLearning #NIST #CybersecurityAwareness

As we step into the age of Multi-modal Large Language Models (#LLMs), we're also stepping into a realm of new challenges. A recent paper sheds light on an unseen aspect of these technological leaps, underscoring the urgency of addressing potential threats in AI security. 𝗞𝗲𝘆 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆𝘀: 1️⃣ Multi-modal LLMs, unlike traditional ones, can interact with multiple types of data (text, images, and audio) and can therefore be exploited through any of these modalities. 2️⃣ Attackers can generate adversarial perturbations that, when blended into images or audio, can manipulate these models into outputting attacker-chosen text or following attacker’s instructions. 3️⃣ These attacks can be stealthier than text attacks as the instruction in the malicious input may not be immediately noticeable by the user. 4️⃣ The researchers demonstrated two types of attacks: targeted-output attacks that make the LLM output any chosen string, and dialog poisoning which leverages the conversation context of LLM-based chatbots. 5️⃣ Notably, the adversarial perturbations do not significantly alter the semantic content of the image or sound, making the attacks harder to detect. Paper: https://lnkd.in/dzDStCDm #aisecurity #aisafety

Crash course on the EU AI Act Part 5: Cybersecurity. (See part 4 of this series here: https://lnkd.in/edQk8a4U). Like any system comprising hardware, software and data, an AI system is vulnerable to security risks, incidents and attacks. But AI systems also present novel cybersecurity risks, including data poisoning or confidentiality attacks (explicitly referred to in the text of the AI Act), model extraction, model inversion, membership inference, backdoors/malware embedded in models (open source or not), code suggestion AI exploited for supply-chain attacks, and many more. For a comprehensive catalogue see MITRE’s ATLAS project https://atlas.mitre.org/. *** Article 15 of the EU AI Act (Parliament draft) requires developers of high risk AI systems to implement security by design and by default. Recital 51 provides additional language. But notice that under the Parliament draft, one of the general principles applicable to *all* AI systems (Article 4a) is “technical robustness and safety,” which requires systems to be “resilient against attempts to alter the use or performance of the AI system so as to allow unlawful use by malicious third parties”. So while high risk AIs must implement more safeguards, security is a principle for all AI. *** Developers of high risk AI systems must implement a risk management system under Article 9. A security risk assessment is part of such risk analysis. Notice that an AI system can be high risk without triggering heightened security issues; and vice versa, it could be low risk but present serious security concerns. *** Responding to the new class of AI-specific cyber risks, the Parliament draft requires providers of high risk AI systems to identify and mitigate “possibly biased outputs Influencing input for future operations (‘feedback loops’) and malicious manipulation of inputs used in learning during operation.” It also requires measures to prevent “attacks trying to manipulate the training dataset (‘data poisoning’), or pre-trained components used in training (‘model poisoning’), inputs designed to cause the model to make a mistake (‘adversarial examples’ or ‘model evasion’),” and more. *** Companies will need to "leverage current cybersecurity practices and procedures, using a combination of existing controls for software systems and AI-model specific measures”. Security standards will play a key role in defining the state of the art. See ENISA presentation, which also addresses the crossroad to the Cyber Resilience Act, here: https://lnkd.in/eAVf5-BH. *** For AI cybersecurity in the context of procurement, see my partners' Kaylee Cox Bankston, CIPP/US  Liza Craig, Jud Welle webcon: https://lnkd.in/eXbZH7Hh.

I am excited to share this new policy brief I co-authored with Landon Klein for the Future of Life Institute (FLI) on risks at the intersection of artificial intelligence and cybersecurity. The brief outlines the primary threats at this intersection, alongside policy recommendations for the US Government to combat these threats. https://lnkd.in/eBTkMwFP Advanced AI systems are becoming increasingly more adept at exploiting cyber vulnerabilities across a wide range of domains. This leaves them open to use by malicious actors to launch zero-day exploits, target critical infrastructure and launch increasingly sophisticated phishing attacks. At the same time, there are hazards to using AI to for cyber-defense, creating vulnerabilities of its own. Finally, there is a dire need to ensure improved information security at the top AI labs in the United States to guard against model theft and misuse.