How to Trust LLM Outputs

I’m jealous of AI Because with a model you can measure confidence Imagine you could do that as a human? Measure how close or far off you are? here's how to measure for technical and non-technical teams For business teams: Run a ‘known answers’ test. Give the model questions or tasks where you already know the answer. Think of it like a QA test for logic. If it can't pass here, it's not ready to run wild in your stack. Ask for confidence directly. Prompt it: “How sure are you about that answer on a scale of 1-10?” Then: “Why might this be wrong?” You'll surface uncertainty the model won't reveal unless asked. Check consistency. Phrase the same request five different ways. Is it giving stable answers? If not, revisit the product strategy for the llm Force reasoning. Use prompts like “Show step-by-step how you got this result.” This lets you audit the logic, not just the output. Great for strategy, legal, and product decisions. For technical teams: Use the softmax output to get predicted probabilities. Example: Model says “fraud” with 92% probability. Use entropy to spot uncertainty. High entropy = low confidence. (Shannon entropy: −∑p log p) Language models Extract token-level log-likelihoods from the model if you have API or model access. These give you the probability of each word generated. Use sequence likelihood to rank alternate responses. Common in RAG and search-ranking setups. For uncertainty estimates, try: Monte Carlo Dropout: Run the same input multiple times with dropout on. Compare outputs. High variance = low confidence. Ensemble models: Aggregate predictions from several models to smooth confidence. Calibration testing: Use a reliability diagram to check if predicted probabilities match actual outcomes. Use Expected Calibration Error (ECE) as a metric. Good models should show that 80% confident = ~80% correct. How to improve confidence (and make it trustworthy) Label smoothing during training Prevents overconfident predictions and improves generalization. Temperature tuning (post-hoc) Adjusts the softmax sharpness to better align confidence and accuracy. Temperature < 1 → sharper, more confident Temperature > 1 → more cautious, less spiky predictions Fine-tuning on domain-specific data Shrinks uncertainty and reduces hedging in model output. Especially effective for LLMs that need to be assertive in narrow domains (legal, medicine, strategy). Use focal loss for noisy or imbalanced datasets. It down-weights easy examples and forces the model to pay attention to harder cases, which tightens confidence on the edge cases. Reinforcement learning from human feedback (RLHF) Aligns the model's reward with correct and confident reasoning. Bottom line: A confident model isn't just better - it's safer, cheaper, and easier to debug. If you’re building workflows or products that rely on AI, but you’re not measuring model confidence, you’re guessing. #AI #ML #LLM #MachineLearning #AIConfidence #RLHF #ModelCalibration

Are your LLM apps still hallucinating? Zep used to as well—a lot. Here’s how we worked to solve Zep's hallucinations. We've spent a lot of cycles diving into why LLMs hallucinate and experimenting with the most effective techniques to prevent it. Some might sound familiar, but it's the combined approach that really moves the needle. First, why do hallucinations happen? A few core reasons: 🔍 LLMs rely on statistical patterns, not true understanding. 🎲 Responses are based on probabilities, not verified facts. 🤔 No innate ability to differentiate truth from plausible fiction. 📚 Training datasets often include biases, outdated info, or errors. Put simply: LLMs predict the next likely word—they don’t actually "understand" or verify what's accurate. When prompted beyond their knowledge, they creatively fill gaps with plausible (but incorrect) info. ⚠️ Funny if you’re casually chatting—problematic if you're building enterprise apps. So, how do you reduce hallucinations effectively? The #1 technique: grounding the LLM in data. - Use Retrieval-Augmented Generation (RAG) to anchor responses in verified data. - Use long-term memory systems like Zep to ensure the model is always grounded in personalization data: user context, preferences, traits etc - Fine-tune models on domain-specific datasets to improve response consistency and style, although fine-tuning alone typically doesn't add substantial new factual knowledge. - Explicit, clear prompting—avoid ambiguity or unnecessary complexity. - Encourage models to self-verify conclusions when accuracy is essential. - Structure complex tasks with chain-of-thought prompting (COT) to improve outputs or force "none"/unknown responses when necessary. - Strategically tweak model parameters (e.g., temperature, top-p) to limit overly creative outputs. - Post-processing verification for mission-critical outputs, for example, matching to known business states. One technique alone rarely solves hallucinations. For maximum ROI, we've found combining RAG with a robust long-term memory solution (like ours at Zep) is the sweet spot. Systems that ground responses in factual, evolving knowledge significantly outperform. Did I miss any good techniques? What are you doing in your apps?

In my AI+UXR workshops, I recommend starting a fresh chat each time you ask the LLM to do a significant task. Why? Because #UXresearch tools need to be reliable, and the more you talk to the LLM, the more that reliability takes a hit. This can introduce unknown errors. This happens for several reasons, but here are a few big (albeit interrelated) ones: 1️⃣ LLMs can get lost even in short multi-turn conversations According to recent research from Microsoft and Salesforce, providing instructions over multiple turns (vs. all at once upfront) can dramatically degrade the output of LLMs.  This is true even for reasoning models like o3 and Deepseek-R1, which “deteriorate in similar ways.” 2️⃣ Past turns influence how the LLM weights different concepts In the workshop, I show a conversation that continuously, subtly references safaris, until the LLM takes a hard turn and generates content with a giraffe in it. Every token influences future tokens, and repeated concepts (even inadvertent ones) can “prime” the model to produce unexpected output. 3️⃣ Every turn is an opportunity for “context poisoning” “Context poisoning” is when inaccurate, irrelevant, or hallucinated information gets into the LLM context, causing misleading results or deviation from instructions. This is sometimes exploited to jailbreak LLMs, but it can happen unintentionally as well. In simple terms, bad assumptions early on are hard to recover from. To avoid these issues, I recommend: 🧩 Starting the conversation from scratch any time you’re doing an important research task (including turning off memory and custom instructions) 🧩 Using a single well-structured prompt when possible 🧩 And always, testing carefully and being alert to errors in LLM output I talk about these issues (and a lot more) in my workshops, and I’m writing about this today because the question was asked by some of my amazing workshop participants. Sign up in my profile to get notified about my next public workshop–or if you’re looking for private, in-house training for your team, drop me a note! #AI #UX #LLM #userresearch

In my work with organizations rolling out AI and generative AI solutions, one concern I hear repeatedly from leaders, and the c-suite is how to get a clear, centralized “AI Risk Center” to track AI safety, large language model's accuracy, citation, attribution, performance and compliance etc. Operational leaders want automated governance reports—model cards, impact assessments, dashboards—so they can maintain trust with boards, customers, and regulators. Business stakeholders also need an operational risk view: one place to see AI risk and value across all units, so they know where to prioritize governance. One of such framework is MITRE’s ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) Matrix. This framework extends MITRE ATT&CK principles to AI, Generative AI, and machine learning, giving us a structured way to identify, monitor, and mitigate threats specific to large language models. ATLAS addresses a range of vulnerabilities—prompt injection, data leakage, malicious code generation, and more—by mapping them to proven defensive techniques. It’s part of the broader AI safety ecosystem we rely on for robust risk management. On a practical level, I recommend pairing the ATLAS approach with comprehensive guardrails - such as: • AI Firewall & LLM Scanner to block jailbreak attempts, moderate content, and detect data leaks (optionally integrating with security posture management systems). • RAG Security for retrieval-augmented generation, ensuring knowledge bases are isolated and validated before LLM interaction. • Advanced Detection Methods—Statistical Outlier Detection, Consistency Checks, and Entity Verification—to catch data poisoning attacks early. • Align Scores to grade hallucinations and keep the model within acceptable bounds. • Agent Framework Hardening so that AI agents operate within clearly defined permissions. Given the rapid arrival of AI-focused legislation—like the EU AI Act, now defunct  Executive Order 14110 of October 30, 2023 (Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence) AI Act, and global standards (e.g., ISO/IEC 42001)—we face a “policy soup” that demands transparent, auditable processes. My biggest takeaway from the 2024 Credo AI Summit was that responsible AI governance isn’t just about technical controls: it’s about aligning with rapidly evolving global regulations and industry best practices to demonstrate “what good looks like.” Call to Action: For leaders implementing AI and generative AI solutions, start by mapping your AI workflows against MITRE’s ATLAS Matrix. Mapping the progression of the attack kill chain from left to right - combine that insight with strong guardrails, real-time scanning, and automated reporting to stay ahead of attacks, comply with emerging standards, and build trust across your organization. It’s a practical, proven way to secure your entire GenAI ecosystem—and a critical investment for any enterprise embracing AI.

The EDPB recently published a report on AI Privacy Risks and Mitigations in LLMs.   This is one of the most practical and detailed resources I've seen from the EDPB, with extensive guidance for developers and deployers. The report walks through privacy risks associated with LLMs across the AI lifecycle, from data collection and training to deployment and retirement, and offers practical tips for identifying, measuring, and mitigating risks.   Here's a quick summary of some of the key mitigations mentioned in the report:   For providers: • Fine-tune LLMs on curated, high-quality datasets and limit the scope of model outputs to relevant and up-to-date information. • Use robust anonymisation techniques and automated tools to detect and remove personal data from training data. • Apply input filters and user warnings during deployment to discourage users from entering personal data, as well as automated detection methods to flag or anonymise sensitive input data before it is processed. • Clearly inform users about how their data will be processed through privacy policies, instructions, warning or disclaimers in the user interface. • Encrypt user inputs and outputs during transmission and storage to protect data from unauthorized access. • Protect against prompt injection and jailbreaking by validating inputs, monitoring LLMs for abnormal input behaviour, and limiting the amount of text a user can input. • Apply content filtering and human review processes to flag sensitive or inappropriate outputs. • Limit data logging and provide configurable options to deployers regarding log retention. • Offer easy-to-use opt-in/opt-out options for users whose feedback data might be used for retraining.   For deployers: • Enforce strong authentication to restrict access to the input interface and protect session data. • Mitigate adversarial attacks by adding a layer for input sanitization and filtering, monitoring and logging user queries to detect unusual patterns. • Work with providers to ensure they do not retain or misuse sensitive input data. • Guide users to avoid sharing unnecessary personal data through clear instructions, training and warnings. • Educate employees and end users on proper usage, including the appropriate use of outputs and phishing techniques that could trick individuals into revealing sensitive information. • Ensure employees and end users avoid overreliance on LLMs for critical or high-stakes decisions without verification, and ensure outputs are reviewed by humans before implementation or dissemination. • Securely store outputs and restrict access to authorised personnel and systems.   This is a rare example where the EDPB strikes a good balance between practical safeguards and legal expectations. Link to the report included in the comments.   #AIprivacy #LLMs #dataprotection #AIgovernance #EDPB #privacybydesign #GDPR

👀 So, you might've heard about the Chevrolet chatbot getting a bit... let's say, 'off-track'. 😅 It's a classic example of "easy to make, hard to master" when it comes to building LLM apps. https://lnkd.in/da_C9R-x 🔧 Sure, tools like LangChain🦜 make it a breeze to whip up an LLM chatbot. But Here's the catch: (Gen)AI security posture is not just a fancy term; it ought to be the backbone of your AI development. 🌐 🛡️ Here's my take on deploying to production a safer RAG app (and avoiding our own Chevy moments): 1️⃣ Prompt Engineering: It's not a silver bullet, but it's a start. Steering the AI away from potentially harmful outputs is crucial and can be done with some protective prompt engineering to the final prompt sent to the LLM. 2️⃣ User Input Scanners: Inspect user generated input that is eventually augmenting your core prompt. This helps to tackle crafty input manipulations. 3️⃣ Prompt Input Scanners:  Double-checking the final prompt before sending it the LLM. Open source tools like @LLM- Guard by Laiyer AI provide a comprehensive suite designed to reinforce the security framework of LLM applications. 4️⃣ Proven Models for RAG: Using tried and tested certain models dedicated to RAG can save you a lot of prompt engineering and coding. 👉 Remember, this list isn't exhaustive, and there's no magic shield for GenAI apps. Think of them as essential AI hygiene practices. They significantly improve your GenAI security posture, laying a stronger foundation for your app. 💬 Bottom line: 👀 The Chevrolet case? Can happen to anyone and It's a wake-up call. BTW It's worth noting the impressive commitment from the LangChain🦜 team. They've really gone all-in, dedicating substantial effort to enhancing safety. Over the past few months, there's been a tremendous push in refactoring their framework, all aimed at providing an infrastructure that's geared towards building more secure and reliable apps Disclaimer: The thoughts and opinions shared here are entirely my own and do not represent those of my employer or any other affiliated organizations.

𝐃𝐢𝐝 𝐲𝐨𝐮 𝐤𝐧𝐨𝐰 𝐋𝐋𝐌 𝐡𝐚𝐥𝐥𝐮𝐜𝐢𝐧𝐚𝐭𝐢𝐨𝐧𝐬 𝐜𝐚𝐧 𝐛𝐞 𝐦𝐞𝐚𝐬𝐮𝐫𝐞𝐝 𝐢𝐧 𝐫𝐞𝐚𝐥-𝐭𝐢𝐦𝐞? In a recent post, I talked about why hallucinations happen in LLMs and how they affect different AI applications. While creative fields may welcome hallucinations as a way to spark out-of-the-box thinking, business use cases don’t have that flexibility. In industries like healthcare, finance, or customer support, hallucinations can’t be overlooked. Accuracy is non-negotiable, and catching unreliable LLM outputs in real-time becomes essential. So, here’s the big question: 𝐇𝐨𝐰 𝐝𝐨 𝐲𝐨𝐮 𝐚𝐮𝐭𝐨𝐦𝐚𝐭𝐢𝐜𝐚𝐥𝐥𝐲 𝐦𝐨𝐧𝐢𝐭𝐨𝐫 𝐟𝐨𝐫 𝐬𝐨𝐦𝐞𝐭𝐡𝐢𝐧𝐠 𝐚𝐬 𝐜𝐨𝐦𝐩𝐥𝐞𝐱 𝐚𝐬 𝐡𝐚𝐥𝐥𝐮𝐜𝐢𝐧𝐚𝐭𝐢𝐨𝐧𝐬? That’s where the 𝐓𝐫𝐮𝐬𝐭𝐰𝐨𝐫𝐭𝐡𝐲 𝐋𝐚𝐧𝐠𝐮𝐚𝐠𝐞 𝐌𝐨𝐝𝐞𝐥 (𝐓𝐋𝐌) steps in. TLM helps you detect LLM errors/hallucinations by scoring the trustworthiness of every response generated by 𝐚𝐧𝐲 LLM.  This comprehensive trustworthiness score combines factors like data-related and model-related uncertainties, giving you an automated system to ensure reliable AI applications. 🏁 The benchmarks are impressive. TLM reduces the rate of incorrect answers from OpenAI’s o1-preview model by up to 20%. For GPT-4o, that reduction goes up to 27%. On Claude 3.5 Sonnet, TLM achieves a similar 20% improvement. Here’s how TLM changes the game for LLM reliability: 1️⃣ For Chat, Q&A, and RAG applications: displaying trustworthiness scores helps your users identify which responses are unreliable, so they don’t lose faith in the AI. 2️⃣ For data processing applications (extraction, annotation, …): trustworthiness scores help your team identify and review edge-cases that the LLM may have processed incorrectly. 3️⃣ The TLM system can also select the most trustworthy response from multiple generated candidates, automatically improving the accuracy of responses from any LLM. With tools like TLM, companies can finally productionize AI systems for customer service, HR, finance, insurance, legal, medicine, and other high-stakes use cases.  Kudos to the Cleanlab team for their pioneering research to advance the reliability of AI. I am sure you want to learn more and use it yourself, so I will add reading materials in the comments!

🚨 Public Service Announcement: If you're building LLM-based applications for internal business use, especially for high-risk functions this is for you. Define Context Clearly ------------------------ 📋 Document the purpose, expected behavior, and users of the LLM system. 🚩 Note any undesirable or unacceptable behaviors upfront. Conduct a Risk Assessment ---------------------------- 🔍 Identify potential risks tied to the LLM (e.g., misinformation, bias, toxic outputs, etc), and be as specific as possible 📊 Categorize risks by impact on stakeholders or organizational goals. Implement a Test Suite ------------------------ 🧪 Ensure evaluations include relevant test cases for the expected use. ⚖️ Use benchmarks but complement them with tests tailored to your business needs. Monitor Risk Coverage ----------------------- 📈 Verify that test inputs reflect real-world usage and potential high-risk scenarios. 🚧 Address gaps in test coverage promptly. Test for Robustness --------------------- 🛡 Evaluate performance on varied inputs, ensuring consistent and accurate outputs. 🗣 Incorporate feedback from real users and subject matter experts. Document Everything ---------------------- 📑 Track risk assessments, test methods, thresholds, and results. ✅ Justify metrics and thresholds to enable accountability and traceability. #psa #llm #testingandevaluation #responsibleAI #AIGovernance Patrick Sullivan, Khoa Lam, Bryan Ilg, Jeffery Recker, Borhane Blili-Hamelin, PhD, Dr. Benjamin Lange, Dinah Rabe, Ali Hasan

Tackling Hallucination in LLMs: Mitigation & Evaluation Strategies As Large Language Models (LLMs) redefine how we interact with AI, one critical challenge is hallucination—when models generate false or misleading responses. This issue affects the reliability of LLMs, particularly in high-stakes applications like healthcare, legal, and education. To ensure trustworthiness, it’s essential to adopt robust strategies for mitigating and evaluating hallucination. The workflow outlined above presents a structured approach to addressing this challenge: 1️⃣ Hallucination QA Set Generation Starting with a raw corpus, we process knowledge bases and apply weighted sampling to create diverse, high-quality datasets. This includes generating baseline questions, multi-context queries, and complex reasoning tasks, ensuring a comprehensive evaluation framework. Rigorous filtering and quality checks ensure datasets are robust and aligned with real-world complexities. 2️⃣ Hallucination Benchmarking By pre-processing datasets, answers are categorized as correct or hallucinated, providing a benchmark for model performance. This phase involves tools like classification models and text generation to assess reliability under various conditions. 3️⃣ Hallucination Mitigation Strategies In-Context Learning: Enhancing output reliability by incorporating examples directly in the prompt. Retrieval-Augmented Generation: Supplementing model responses with real-time data retrieval. Parameter-Efficient Fine-Tuning: Fine-tuning targeted parts of the model for specific tasks. By implementing these strategies, we can significantly reduce hallucination risks, ensuring LLMs deliver accurate and context-aware responses across diverse applications. 💡 What strategies do you employ to minimize hallucination in AI systems? Let’s discuss and learn together in the comments!

Building Safer AI Starts with Guardrails. Here's How to Add Them. LLM outputs are powerful, but without control, they can be risky. Guardrails AI helps implement safety, compliance, and accuracy at every step of AI interaction. 1. What Is Guardrails AI? It’s an open-source framework designed to validate and monitor LLM inputs and outputs. It applies rules like detecting PII, filtering hallucinations, and blocking jailbreaks to ensure responsible use. 2. Why Use It? From preventing harmful replies to ensuring regulatory compliance, Guardrails AI reduces risk and improves the quality of AI responses especially in sensitive applications like healthcare, support, and enterprise tools. 3. How It Works Install via pip, configure guards, validate inputs and outputs in real-time, and enforce schema rules or safety policies as needed, all using structured workflows. 4. What It Offers Prebuilt validators for tone, bias, PII, and more. Plus, tools for structured outputs, real-time checks, and custom rule enforcement for specific needs. 5. Where It’s Used Perfect for customer support, educational bots, compliant healthcare assistants, and content tools where trust and safety are non-negotiable. Guardrails AI isn't just an extra layer, it's your AI’s safety net. Save this if you’re working with LLMs in regulated or public-facing environments.