Top Owasp Risks for Llms

Live from a long flight home: I did some heavy reading so you don’t have to 😏 → A Spring 2025 overview of top AI Security Risks for Enterprise. 1. Prompt injection & jailbreaking - Bypass the model’s guardrails - Indirect injection is on the rise using PDFs, emails, etc. - Manipulate it, leak training data: customer info, IP, … 2. Model/supply chain compromise - Devs often use pre-trained AI models from 3rd parties - Hidden backdoor in a model = you’re compromised! - Ex: Sleepy Pickle, with malicious code hidden in the model, and triggered once deployed 3. Poisoned datasets - A poisoned dataset can make a model misbehave - Ex: fail to detect fraud, or misclassify malware - Cheap! As little as $60 to poison a dataset like LAION 4. Extremely convincing deepfakes - Think perfect (fake) videos of your CTO asking for a network policy change - Crafted with public samples of the CTO’s voice/video - Leads to a security breach 5. Agentic AI threats - AI agents can have vast powers on a system - But they can be compromised by new kinds of malware - That malware can write its own code and “learn” to break a system over time ---- It doesn’t mean we need to slow down on AI. It’s important however to: - Educate teams - Put the right guardrails in place - Manage risk at every point of the AI lifecycle - Leverage frameworks such as OWASP/MITRE Annnnddd.... Leveraging a solution such as Cisco AI Defense can really help manage AI risk: - Get full visibility across AI apps, models, etc. - Define & enforce granular policy around the use of AI - Validate models before they go in prod (including through algorithmic jailbreaking) - Protect AI apps during runtime Anand, Manu, DJ and all other AI security gurus here: what did I forget?

OWASP LLM Top 10 for LLM's 2025 🤖 🔟 The OWASP® Foundation Top 10 for LLM Applications projects just dropped its first major update since 2023. This is an excellent staple for the community to rally around when it comes to securing GenAI and LLM systems. Some key aspects of this latest release: - New and updated risks based on broader worldwide contributions from researchers and practitioners - Addition of recommendations for securing Retrieval-Augmented Generation (RAG) methods - Expanding risks and mitigations associated with Prompts, including Injection and System Prompt Leakage - A focus on "excessive agency" as we see a TON of excitement and advancement in the area of agentic architectures and Agentic AI It also is rich in reference links of research, publications, and real-world incidents, as well as cross-walked to MITRE'S ATLAS framework I recommend folks give it a read and bookmark it to keep it on hand. As always, amazing work by Steve Wilson and Ads Dawson and the broader OLWASP LLM Top 10 team! #ciso #ai #cyber

🔐 Just dropped a comprehensive 𝑨𝑰 𝒔𝒆𝒄𝒖𝒓𝒊𝒕𝒚 𝒄𝒉𝒆𝒂𝒕𝒔𝒉𝒆𝒆𝒕! After diving deep into the world of LLM security with customers (and making plenty of mistakes along the way 😅), I've created the resource I wish I had when starting out. 🔖 My latest blog breaks down the 𝐎𝐖𝐀𝐒𝐏 𝐓𝐨𝐩 10 𝐟𝐨𝐫 𝐋𝐋𝐌𝐬 𝐚𝐧𝐝 𝐆𝐞𝐧𝐀𝐈 in plain English - mapping exactly WHERE in your application architecture each security measure needs to be implemented. 📍 Whether you're building your first AI app or hardening existing systems, this visual guide shows you the exact weak points attackers are targeting right now. 👉 Check out the full breakdown at https://lnkd.in/dpCi4Ws6 and let me know what security challenges you're facing with your AI projects! #AISecurity #OWASP #MachineLearning #CyberSecurity #GenAI #LLMSecurity

🚨 🤯 How attackers can jailbreak LLMs and leak system prompts—meet PLeak.‼️ System prompt leakage is rapidly emerging as one of the most critical threats in GenAI security. In Trend’s latest research, Karanjot Singh Saggu and Anurag Das introduce PLeak, an algorithmic method that auto-generates adversarial prompts to exfiltrate hidden system instructions—revealing everything from internal rules to tokens and file paths. PLeak aligns with major risk categories from MITRE and OWASP® Foundation: • MITRE ATLAS – LLM Meta Prompt Extraction • MITRE ATLAS – Privilege Escalation • MITRE ATLAS – Credential Access • OWASP LLM07 – System Prompt Leakage • OWASP LLM06 – Excessive Agency In tests across major LLMs, PLeak achieved high success rates, even when not optimized for the target model: • GPT-4 • GPT-4o • Claude 3.5 Sonnet v2 • Claude 3.5 Haiku • Mistral Large • Mistral 7B • Llama 3.2 3B • Llama 3.1 8B • Llama 3.3 70B • Llama 3.1 405B Shockingly, success was even higher on #Mistral models than the #Llama models PLeak was trained on—showing strong cross-model transferability. Organizations deploying LLMs must take proactive steps: • Train with adversarial examples • Detect jailbreak prompts using classifiers • Enforce access control for AI applications Check out the 🛡️ Security for AI Blueprint to help shape your security layer to protect your AI Applications: https://lnkd.in/gPBXFdsZ Trend Micro is also collaborating with OWASP® Foundation and MITRE ATLAS to help shape a secure AI future. Don’t let GenAI innovation outpace your defenses. Read the full research and see PLeak in action: https://lnkd.in/gsFXefeK #GenAI #LLM #AIsecurity #PromptLeakage #PLeak #TrendMicro #ThreatResearch

𝑻𝒉𝒆 𝒏𝒆𝒘 𝑶𝑾𝑨𝑺𝑷 𝑻𝒐𝒑 𝟏𝟎 𝑳𝑳𝑴 𝒇𝒐𝒓 𝟐𝟎𝟐𝟓 𝒊𝒔 𝒉𝒆𝒓𝒆! 👏 OWASP Top 10 For Large Language Model Applications & Generative AI has proven to be a grounding representative of the things we are witnessing, discussing and working on in AI Security and Safety all through the year of 2024. Now, the OWASP team has come up with a newer version of the document for 2025. So what's changed?💡 A couple of new and exciting additions - more emphasis on agentic security concerns, real world impacts through misinformation generation and supply chain security lapses, wider outcomes of unbounded consumption and possible attacks through vectors and embeddings! But the most interesting one for me to read was the inclusion of 𝐒𝐘𝐒𝐓𝐄𝐌 𝐏𝐑𝐎𝐌𝐏𝐓 𝐋𝐄𝐀𝐊𝐀𝐆𝐄. Why so? I’ll tell you - All through 2024, for all the customers that we have worked with - for every single one of them, we were able to leak their system prompts. Either through our open sourced automated tool 𝐖𝐡𝐢𝐬𝐭𝐥𝐞𝐛𝐥𝐨𝐰𝐞𝐫 (which has been built specifically for this purpose) or our automated red-teaming product. While, system prompt leak in itself is not a high impact risk but it gives the attacker the best insight into the conditioning and sometimes the architecture of the AI application and thus can help in executing higher impact attacks - like guardrails bypass, privilege escalation, credentials leakage and more. Thus, for all those customers that had some form of sensitive information in their system prompt our first suggestion to them was to remove anything sensitive from it and push these things deeper into the internal layers or build separate modules for it. For sure, there are ways using which you can harden your system prompt and thus ensure the edge gained from the efforts and engineering hours put into perfecting it are not in vain - it’s impossible (at least at the moment of writing this) for applications to prevent a system prompt leak in a 100% assured manner. Thus, it should ideally be something that does not cost you your job if some malicious actor is able to get it out! I’ll be writing more on this covering the other changes and topics that caught my attention in the new OWASP Top 10 LLM from an industry and practical POV, in detail soon - so stay tuned! #owaspTop10LLM #AI #security #safety #responsibleAI