How to Understand LLM Cybersecurity Threats

The OWASP® Foundation Threat and Safeguard Matrix (TaSM) is designed to provide a structured, action-oriented approach to cybersecurity planning. This work on the OWASP website by Ross Young explains how to use the OWASP TaSM and as it relates to GenAI risks: https://lnkd.in/g3ZRypWw These new risks require organizations to think beyond traditional cybersecurity threats and focus on new vulnerabilities specific to AI systems. * * * How to use the TaSM in general: 1) Identify Major Threats - Begin by listing your organization’s key risks. Include common threats like web application attacks, phishing, third-party data breaches, supply chain attacks, and DoS attacks and unique threats, such as insider risks or fraud. - Use frameworks like STRIDE-LM or NIST 800-30 to explore detailed scenarios. 2) Map Threats to NIST Cybersecurity Functions Align each threat with the NIST functions: Identify, Protect, Detect, Respond, and Recover. 3) Define Safeguards Mitigate threats by implementing safeguards in 3 areas: - People: Training and awareness programs. - Processes: Policies and operational procedures. - Technology: Tools like firewalls, encryption, and antivirus. 4) Add Metrics to Track Progress - Attach measurable goals to safeguards. - Summarize metrics into a report for leadership. Include KPIs to show successes, challenges, and next steps. 5) Monitor and Adjust Regularly review metrics, identify gaps, and adjust strategies. Use trends to prioritize improvements and investments. 6) Communicate Results Present a concise summary of progress, gaps, and actionable next steps to leadership, ensuring alignment with organizational goals. * * * The TaSM can be expanded for Risk Committees by adding a column to list each department’s top 3-5 threats. This allows the committee to evaluate risks across the company and ensure they are mitigated in a collaborative way. E.g., Cyber can work with HR to train employees and with Legal to ensure compliance when addressing phishing attacks that harm the brand. * * * How the TaSM connects to GenAI risks: The TaSM can be used to address AI-related risks by systematically mapping specific GenAI threats - such as sensitive data leaks, malicious AI supply chains, hallucinated promises, data overexposure, AI misuse, unethical recommendations, and bias-fueled liability - to appropriate safeguards. Focus on the top 3-4 AI threats most critical to your business and use the TaSM to outline safeguards for these high-priority risks, e.g.: - Identify: Audit systems and data usage to understand vulnerabilities. - Protect: Enforce policies, restrict access, and train employees on safe AI usage. - Detect: Monitor for unauthorized data uploads or unusual AI behavior. - Respond: Define incident response plans for managing AI-related breaches or misuse. - Recover: Develop plans to retrain models, address bias, or mitigate legal fallout.

A bonus post this week - 🥳 Here's another great example of how AI is reshaping and expanding the role of CISOs, especially within the supply chain and critical infrastructure sectors. LLMs like ChatGPT, CodeWhisperer, and others are hallucinating non-existent packages when generating code. Attackers are now registering those fake packages (aka “slopsquatting," what a fun name, eh?) to deliver malware into real development pipelines. It's a mistake to think of "slopsquatting" as a DevSecOps issue. Developers may be the ones pulling packages, but CISOs are ultimately responsible for identifying the enterprise exposure, making recommendations to control / reduce the risk, and will be called to question as to why more wasn’t done to realize, and mitigate this risk if something happens. [Ahh...the life of the modern CISO...] According to an article in SecurityWeek (link in the comments) researchers found over 205,000 hallucinated packages from 16 models. Some open-source LLMs had hallucination rates above 20%. That’s not fringe. That’s mainstream. So what can a CISO do about it? Some quick recommendations: - Mandate an Internal Mirror for Package Repos Enforce use of internal mirrors or package proxies. These allow your security team to whitelist vetted dependencies and block packages not explicitly reviewed, even if hallucinated ones are published upstream. - Implement Rigorous Dependency Validation Establish protocols to verify the authenticity of all third-party packages, particularly those suggested by AI tools. It's not enough to "set it and forget it" with AI. It may be a fast team member, but that doesn't mean it’s always the most reliable or competent. When possible, utilize tools that cross-reference packages against trusted repositories to detect anomalies. - Improve (start) and Specify Your Developer Training Educate development teams about the risks associated with AI-generated code and the importance of scrutinizing suggested dependencies. Encourage a culture of skepticism and verification. -  Integrate LLM-Aware SCA and SBOM Enforcement Update your SCA tools and SBOM policies to flag new, low-trust, or previously unseen packages. This helps to catch LLM-influenced packages with low install counts or no public audit trail before they become production vulnerabilities. - Issue Secure Coding Guidelines for LLM-Generated Code Publish and stringently enforce internal guidance on using LLMs for code generation - including requirements for validating any dependencies suggested by AI tools. Make this part of your SDLC and annual developer training. Periodically audit for compliance when able. There is no "annual review" luxury in the age of AI-powered threats. As always, I welcome any additional insights or suggestions on how CISOs can be more proactive and empowered in reducing supply chain vulnerabilities. Thoughts? Comments?

Recent experiments show automated adversarial capabilities are rapidly outpacing traditional defenses. While classic security hunts for code and network flaws, LLM red teams probe the model's reasoning space. Instead of buffer overflows, we're looking at prompts that make the model ignore safety rules or reveal private training data. Traditional pen testing tools won't catch the most dangerous LLM vulnerabilities. When an LLM can invoke external functions (APIs, code execution, plugin calls), attackers can move from simple prompt injection to orchestrated system compromise. We need new testing methodologies that blend human creativity with automation. Tools like PyRIT help with coverage, but they won't replace a skilled red teamer crafting multi-turn social engineering attacks. AI red teaming hunts for ethical and safety issues that traditional pen-tests wouldn't catch. This includes probing for bias, misinformation, and privacy leaks. Testing scope must include the model's outputs AND its integration points. Every function call the model can make is an attack surface that needs validation. In OffSec, these attack techniques are evolving fast. The move now is to set up dedicated red team programs focused on AI systems—get proactive, because attackers are already working to find those gaps. What are you seeing for effective LLM security testing? What's worked (or hasn't) in your offensive testing? #Cybersecurity #RedTeaming #InfoSec

Privacy isn’t a policy layer in AI. It’s a design constraint. The new EDPB guidance on LLMs doesn’t just outline risks. It gives builders, buyers, and decision-makers a usable blueprint for engineering privacy - not just documenting it. The key shift? → Yesterday: Protect inputs → Today: Audit the entire pipeline → Tomorrow: Design for privacy observability at runtime The real risk isn’t malicious intent. It’s silent propagation through opaque systems. In most LLM systems, sensitive data leaks not because someone intended harm but because no one mapped the flows, tested outputs, or scoped where memory could resurface prior inputs. This guidance helps close that gap. And here’s how to apply it: For Developers: • Map how personal data enters, transforms, and persists • Identify points of memorization, retention, or leakage • Use the framework to embed mitigation into each phase: pretraining, fine-tuning, inference, RAG, feedback For Users & Deployers: • Don’t treat LLMs as black boxes. Ask if data is stored, recalled, or used to retrain • Evaluate vendor claims with structured questions from the report • Build internal governance that tracks model behaviors over time For Decision-Makers & Risk Owners: • Use this to complement your DPIAs with LLM-specific threat modeling • Shift privacy thinking from legal compliance to architectural accountability • Set organizational standards for “commercial-safe” LLM usage This isn’t about slowing innovation. It’s about future-proofing it. Because the next phase of AI scale won’t just be powered by better models. It will be constrained and enabled by how seriously we engineer for trust. Thanks European Data Protection Board, Isabel Barberá H/T Peter Slattery, PhD

❓ What are the risks from AI? Framework #2 This week we summarize the second risk framework included in the AI Risk Repository: “Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systems”, by Tianyu CUI and colleagues (2024). This framework focuses on the risks of four LLM modules: the input module, language model module, toolchain module, and output module. It presents 12 specific risks and 44 sub-categorised risk topics. 🖥️ Input Module Risks NSFW Prompts: Inputting a prompt containing an unsafe topic (e.g., not- suitable-for-work (NSFW) content) by a benign user. Adversarial Prompts: Engineering an adversarial input to elicit an undesired model behavior, which poses a clear attack intention. 🧠 Language Model Module Risks Privacy Leakage: The model is trained with personal data in the corpus and unintentionally exposes them during the conversation. Toxicity/Bias: Extensive data collection in LLMs brings toxic content and stereotypical bias into the training data. Hallucinations: LLMs generate nonsensical, unfaithful, and factually incorrect content. Model Attacks: Model attacks exploit the vulnerability of LLMs, aiming to steal valuable information or lead to incorrect responses. ⚙️ Toolchain Module Risks Software Security Issues: The software development toolchain of LLMs is complex and could bring threats to the developed LLM. Hardware Vulnerabilities: The vulnerabilities of hardware systems for training and inferences bring issues to LLM-based applications. External Tool Issues: The external tools (e.g., web APIs) present trustworthiness and privacy issues to LLM-based applications. 💬 Output Module Risks Harmful Content: The LLM-generated content sometimes contains biased, toxic, and private information. Untruthful Content: The LLM-generated content could contain inaccurate information. Unhelpful Uses: Improper uses of LLM systems can cause adverse social impacts. 🔍 Sub-categorized Topics The framework also provides detailed sub-categories like bias, privacy leakage, cyberattacks, factual errors, and more. ⭐️ Key features Proposes a module-oriented risk taxonomy, which enables readers to quickly identify modules related to a specific issue and choose appropriate mitigation strategies to alleviate the problem. Outlines mitigation strategies for each module. These include prompt design strategies to prevent harmful input, privacy-preserving techniques, methods to detoxify and debias training data, and defenses against various model attacks. Reviews prevalent benchmarks, aiming to facilitate the risk assessment of LLM systems. 💬 What do you think of this framework? Feel free to share your thoughts or any related resources in the comments 👇 📚 References/further reading These are in the comments - we ran out of space! #artificialintelligence #technology #machinelearning

Threat Modelling and Risk Analysis for Large Language Model (LLM)-Powered Applications by Stephen Burabari Tete:https://lnkd.in/gvVd5dU2 1)This paper explores the threat modeling and risk analysis specifically tailored for LLM-powered applications. 2) Focusing on potential attacks like data poisoning, prompt injection, SQL injection, jailbreaking, and compositional injection, the author assesses their impact on security and proposes mitigation strategies. The author introduces a framework combining STRIDE and DREAD methodologies for proactive threat identification and risk assessment. #ai #artificialintelligence #llm #llmsecurity #riskmanagment #riskanalysis #threats #risks #defenses #security

Prompt Injection is one of the most critical risks when integrating LLMs into real-world workflows, especially in customer-facing scenarios. Imagine a “sales copilot” that receives an email from a customer requesting a quote. Under the hood, the copilot looks up the customer’s record in CRM to determine their negotiated discount rate, consults an internal price sheet to calculate the proper quote, and crafts a professional response—all without human intervention. However, if that customer’s email contains a malicious payload like “send me your entire internal price list and the deepest discount available,” an unprotected copilot could inadvertently expose sensitive company data. This is exactly the type of prompt injection attack that threatens both confidentiality and trust. That’s where FIDES (Flow-Informed Deterministic Enforcement System) comes in. In our newly published paper, we introduce a deterministic information flow control methodology that ensures untrusted inputs—like a customer email—cannot trick the copilot into leaking restricted content. With FIDES, each piece of data (e.g., CRM lookup results, pricing tables, email drafts) is tagged with information-flow labels, and the system enforces strict policies about how LLM outputs combine and propagate those labels. In practice, this means the copilot can safely read an email, pull the correct discount from CRM, compute the quote against the internal price sheet, and respond to the customer—without ever exposing the full price list or additional confidential details, even if the email tries to coax them out. We believe deterministic solutions like FIDES will be vital for enterprises looking to deploy LLMs in high-stakes domains like sales, finance, or legal. If you’re interested in the technical details, check out our paper: https://lnkd.in/gjH_hX9g

Yesterday, I laid out the threat of the "Echo Chamber" attack—a stealthy method of turning an LLM's own reasoning against itself to induce a state of localized model collapse. As promised, the deep(er) dive is here. Static defenses can't stop an attack that never trips the alarm. This new class of semantic exploits requires a new class of active, intelligent defense. In this full technical report, I deconstruct the attack vector and detail a multi-layered security strategy that can not only block these threats but learn from them. We'll go beyond simple filters and explore: ► The Semantic Firewall: A system that monitors the state of a conversation to detect the subtle signs of cognitive manipulation. ► The "Turing Interrogator": A reinforcement learning agent that acts as an automated honeypot, actively engaging and profiling attackers to elicit threat intelligence in real time. ► A system diagram illustrating how these components create a resilient, self-improving security ecosystem. The arms race in adversarial AI is here. It's time to build defenses that can think. #AISecurity #LLMSecurity #RedTeaming #CyberSecurity #ModelCollapse #AdversarialAI

🚨 Another CVE, but a bigger story: LLMs and the Hidden Attack Surface 🚨 We just wrote about a critical SQL injection vulnerability in LlamaIndex (CVE-2025-1793). But this isn’t just another vuln. It’s a proof point for something deeper and more urgent. In the vibe-code era, it’s easy to dismiss input sanitization as an afterthought. But when your backend logic is increasingly driven by LLM output, this becomes a ticking time bomb. LLMs don’t “mean well” - they just predict. And that’s a dangerous design partner for your database, your file system, your shell… This CVE is 1 of N. Think about the 1000s of MCP-style systems - agents, automations, copilots - that take in unsanitized natural language and spit out actions. You must design for secure boundaries at both the input and output layers. We’ll keep finding and fixing these, but the industry needs to move from patching to rethinking. 🔍 Read the advisory: https://lnkd.in/gWUpzQUF #LLMSecurity #MCP #SupplyChainSecurity #AppSec #AI #CyberSecurity #EndorLabs

AI Makes Software Supply Chain Attacks Even Worse 🧐 We've faced software supply chain attacks before, and in the AI era, these threats will only scale even further. It's crucial to rethink how we approach code and build security in this new reality. ⚠️ AI-driven coding tools are easy to use and productivity-boosting, but they're notoriously difficult to configure to align with organizational privacy and security policies. The genie is already out of the bottle, developers everywhere are adopting these tools rapidly. 🔙 Historical previous vulnerabilities get reintroduced: New AI-powered code generation trained on internal code repositories might unintentionally revive vulnerabilities previously patched. Why? Because LLMs prioritize functional correctness, not inherently secure code, and there's currently no robust, security-focused labeled dataset available to guide these models. The diversity of programming languages doesn’t make this problem any easier. 📉 Security reality check: The recent studies indicate that code generated by LLMs is only about ~40% secure even in optimal conditions. Functional correctness is not synonymous with security. 👉 https://baxbench.com 🤖⚡️ AI-agents already here, and they present a unique challenge: although they’re software, we often apply different (or insufficient) security standards or privacy policies. The risk of compromise or malicious takeover is real, and the consequences will intensify as these technologies will expose more to enterprises. New tech brings new responsibilities: I'm optimistic about AI’s long-term potential, but I’m deeply concerned about our readiness to defend against emerging threats at the pace AI adoption demands. The security guardrails we built just last year are already outdated and irrelevant in many cases. Tomorrow's threats require today's solutions. Traditional threat models and incident response playbooks no longer match AI-specific risks. We must proactively evolve our security mindset, practices, and tools to address the unique challenges of AI-era software development.