How to Ensure Trust in AI Medical Outputs

From Toys to Tools: Making Generative AI a True Asset in Healthcare Despite big opportunities for genAI in healthcare, there’s a huge adoption gap at the moment…hard to know exactly how big but there are hundreds of approved applications and only a handful in use in most health systems today. There are lots of very good reasons for this: safety, security, privacy among the many. Right now, many genAI applications in healthcare get great traction for a limited period and then fall into disuse…to me that’s a clear sign that these tools are not yet enabling productivity. It’s a nice to have, not a must have. So how do we move from “toys” to real efficiency-optimizing “tools"? First, why isn’t AI driving real productivity in healthcare yet? 3 primary reasons (there are more!): 1. Accuracy & Hallucination Risks – A single incorrect recommendation can have life-or-death consequences. HC is appropriately cautious here and doesn’t have the monitoring in place to guard against this. Because of these risks, AI today still needs a lot of human oversight and correction. 2. Lack of Workflow Integration – Most AI tools operate outside of clinicians’ natural workflows, forcing extra steps instead of removing them. 3. Trust & Adoption Barriers – Clinicians are understandably skeptical. If an AI tool slows them down or introduces errors, they will abandon it. How Can We Make AI a True Tool for Healthcare? 3 main moves we need to make: 1. Embed Trust & Explainability AI can’t just generate outputs—it has to show its reasoning (cite sources, flag uncertainty, allow inspection). And, it needs to check itself using other gen & non-genAI tools to double & triple check the outcomes in areas of high sensitivity. 2. Seamless Workflow Integration For AI to become truly useful, it must integrate with existing workflows, Auto-populating existing tools (like the EHR) and completing "last mile" steps like communicating with patients. 3. Reducing the Burden on our Workforce, Not Adding to It The tech is not enough…at-the-elbow change management will be needed to ensure human adoption and workflow adaptation and we will need to track the impact of these tools on the workforce and our patient communities. The Future: AI That Feels Invisible, Yet Indispensable Right now, genAI in healthcare is still early—full of potential but struggling to deliver consistent, real-world value. The best AI solutions of the future will be those that:  ✅ Enhance—not replace—clinicians’ expertise ✅ Are trusted because they are explainable and reliable ✅ Reduce administrative burden, giving providers more time for patients ✅ Integrate seamlessly into existing healthcare workflows Ultimately, if we build a successful person-tech interaction, the best AI won't be a novelty but an essential tool to enable us to see where our workflows are inefficient and allow us to change them effectively. What do you think? What’s the biggest barrier to making AI truly useful in healthcare?

🩺 “The scan looks normal,” the AI system says. The doctor hesitates. Will the clinician trust the algorithm? And perhaps most importantly—should they? We are entering an era where artificial intelligence will be woven into the fabric of healthcare decisions, from triaging patients to predicting disease progression. The potential is breathtaking: earlier diagnoses, more efficient care, personalized treatment plans. But so are the risks: opaque decision-making, inequitable outcomes, and the erosion of the sacred trust between patient and provider. The challenge is no longer just about building better AI. It’s about building better ways to decide if—and how—we should use it. That’s where the FAIR-AI framework comes in. Developed through literature reviews, stakeholder interviews, and expert workshops, it offers healthcare systems a practical, repeatable, and transparent process to: 👍 Assess risk before implementation, distinguishing low, moderate, and high-stakes tools. 👍 Engage diverse voices, including patients, to evaluate equity, ethics, and usefulness. 👍 Monitor continuously, ensuring tools stay aligned with their intended use and don’t drift into harm. 👍 Foster transparency, with plain-language “AI labels” that demystify how tools work. FAIR-AI treats governance not as a barrier to innovation, but as the foundation for trust—recognizing that in medicine, the measure of success isn’t how quickly we adopt technology, but how wisely we do it. Because at the end of the day, healthcare isn’t about technology. It’s about people. And people deserve both the best we can build—and the safeguards to use it well. #ResponsibleAI #HealthcareInnovation #DigitalHealth #PatientSafety #TrustInAI #HealthEquity #EthicsInAI #FAIRAI #AIGovernance #HealthTech

Superhuman AI agents will undoubtedly transform healthcare, creating entirely new workflows and models of care delivery. In our latest paper from  Google DeepMind Google Research Google for Health, "Towards physician-centered oversight of conversational diagnostic AI," we explore how to build this future responsibly. Our approach was motivated by two key ideas in AI safety: 1. AI architecture constraints for safety: Inspired by concepts like 'Constitutional AI,' we believe systems must be built with non-negotiable rules and contracts (disclaimers aren’t enough). We implemented this using a multi-agent design where a dedicated ‘guardrail agent’ enforces strict constraints on our AMIE AI diagnostic dialogue agent, ensuring it cannot provide unvetted medical advice and enabling appropriate human physician oversight. 2. AI system design for trust and collaboration: For optimal human-AI collaboration, it's not enough for an AI's final output to be correct or superhuman; its entire process must be transparent, traceable and trustworthy. We implemented this by designing the AI system to generate structured SOAP notes and predictive insights like diagnoses and onward care plans within a ‘Clinician Cockpit’ interface optimized for human-AI interaction. In a comprehensive, randomized OSCE study with validated patient actors, these principles and design show great promise: 1. 📈 Doctors time saved for what truly matters: Our study points to a future of greater efficiency, giving valuable time back to doctor. The AI system first handled comprehensive history taking with the patient. Then, after the conversation, it synthesized that information to generate a highly accurate draft SOAP note with diagnosis - 81.7% top-1 diagnostic accuracy 🎯 and > 15% absolute improvements over human clinicians - for the doctor’s review. This high-quality draft meant the doctor oversight step took around 40% less time ⏱️ than a full consultation performed by a PCP in a comparable prior study. 2. 🧑⚕️🤝 A framework built on trust: The focus on alignment resulted in a system preferred by everyone. The architecture guardrails proved highly reliable with the composite system deferring medical advice >90% of the time. Overseeing physicians reported a better experience with the AI ✅ compared to the human control groups, and (actor) patients strongly preferred interacting with AMIE ⭐, citing its empathy and thoroughness. While this study is an early step, we hope its findings help advance the conversation on building AI that is not only superhuman in capabilities but also deeply aligned with the values of the practice of medicine. Paper - https://lnkd.in/gTZNwGRx Huge congrats to David Stutz Elahe Vedadi David Barrett Natalie Harris Ellery Wulczyn Alan Karthikesalingam MD PhD Adam Rodman Roma Ruparel, MPH Shashir Reddy Mike Schäkermann Ryutaro Tanno Nenad Tomašev S. Sara Mahdavi Kavita Kulkarni Dylan Slack for driving this with all our amazing co-authors.

One of the biggest challenges with AI-generated content is trust. If you don’t know where the information is coming from, how can you rely on it, especially in healthcare? That’s why Citations are a core feature in CareCo. Every piece of generated content, whether it’s a Call Guide, Task List, or Patient Summary, is always backed by citations linking directly to the exact source data that informed it. Here’s why Citations are critical: AI Transparency: You’ll always know where insights come from—whether it’s a past conversation, a diagnosis, a care plan, or another data source. Builds Trust: Care coordinators, managers, and providers can confidently use AI-driven recommendations, knowing they are grounded in real patient data. Accountability & Accuracy: Citations eliminate guesswork, making it easy to verify and validate AI-generated content. AI should amplify human decision-making, not replace it. Citations ensure that every care coordinator using CareCo has full visibility into what’s driving their workflow. Find out more at https://careco.ai/ Soujanya (Chinni) Pulluru MD Jay Hoffman Jenn Brooks Kaluza

Should you blindly trust AI? Most teams make a critical mistake with AI - we accept its answers without question, especially when it seems so sure. But AI confidence ≠ human confidence. Here’s what happened: The AI system flagged a case of a rare autoimmune disorder. The doctor, trusting the result, recommended an aggressive treatment plan. But something felt off. When I was called in to review, we discovered the AI had misinterpreted an MRI anomaly. The patient had a completely different condition - one that didn't require that aggressive treatment. One wrong decision, based on misplaced trust, could’ve caused real harm. To prevent this amid the integration of AI into the workforce, I built the “acceptability threshold” framework. Here’s how it works: This framework is copyrighted: © 2025 Sol Rashidi. All rights reserved. 1. Measure how accurate humans are at a task (our doctors were 93% accurate on CT scans) 2. Use that as our minimum threshold for AI. 3. If AI's confidence falls below this human benchmark, a person reviews it. This approach transformed our implementation and prevented future mistakes. The best AI systems don't replace humans - they know when to ask for human help. What assumptions about AI might be putting your projects at risk?

Presentations of the FDA Digital Health Advisory Committee Meeting on Generative AI-Enabled Devices: Enhancing Postmarket Monitoring and Management Approaches for Managing Changes in AI-Enabled Medical Devices Jessica Paulsen, Associate Director at FDA, presented the regulatory framework and methodologies for managing changes in AI-enabled medical devices. She emphasized the importance of a Total Product Life Cycle (TPLC) approach to ensure continuous safety and effectiveness. She outlined two critical mechanisms: postmarket monitoring through special controls and Predetermined Change Control Plan (PCCP). She explained how postmarket performance plans mitigate risks like bias and data quality issues. Reimagining Regulatory Oversight for AI in Healthcare Christopher Longhurst, Chief Clinical and Innovation Officer at UC San Diego Health, proposed significant changes to the regulatory framework for AI in healthcare. He argued that the FDA should reconsider the 510(k) pathway for AI algorithms and strengthen postmarketing surveillance. Additionally, he emphasized the shared responsibility of healthcare organizations in local testing and monitoring. Postmarket Performance Monitoring in Radiology: A Clinical Perspective Nina Kottler, MD, MS, FSIIM, Associate Chief Medical Officer for Clinical AI at Radiology Partners, shared her experience deploying AI tools in radiology. She focused on continuous validation and error mitigation in generative AI, particularly through expert-in-the-loop systems. She concluded that while generative AI holds promise, it requires expert oversight and robust validation processes. A Patient-Centric Approach to Postmarket Performance Monitoring Grace Cordovano, PhD, BCPA, a board-certified Patient Advocate, presented the patient and caregiver perspective on AI in healthcare. She emphasized the importance of co-creating postmarket monitoring frameworks with patients, ensuring transparency and trust. She concluded by stressing the need for clear escalation paths for reporting AI-related concerns, similar to those available for other aspects of healthcare. Generative AI in Healthcare: Challenges in Postmarket Monitoring Dale Webster, Director of Health AI Research at Google, discussed the unique challenges of evaluating generative AI in healthcare. He emphasized that while the AI life cycle remains consistent, generative models require new evaluation frameworks. He presented Google's postmarket monitoring approach for imaging AI, which includes sampling, human review, and dashboard analyses. However, he acknowledged that evaluating generative AI’s textual outputs is far more complex. Existing metrics for predictive AI, such as sensitivity and specificity, are inadequate for assessing the infinite possible outputs of generative models. Video Link: https://lnkd.in/eF9CfaSr #GenerativeAI #LarageLanguageModels #LLMs #AIinHealthcare #Regulation #FDA #TPLC #Compliance

FDA Calls for Greater Transparency and Bias Mitigation in AI Medical Devices: ⚖️The recently issued US FDA draft guidance emphasizes transparency in AI device approvals, recommending detailed disclosures on data sources, demographics, blind spots, and biases ⚖️ Device makers should outline validation data, methods, and postmarket performance monitoring plans to ensure ongoing accuracy and reliability ⚖️ The guidance highlights the need for data diversity to minimize bias and ensure generalizability across populations and clinical settings ⚖️ Recommendations include using “model cards” to provide clear, concise information about AI models and their updates ⚖️ The FDA proposes manufacturers submit plans for updating and maintaining AI models without requiring new submissions, using pre-determined change control plans (PCCP) ⚖️ Concerns about retrospective-only testing and site-specific biases in existing AI devices highlight the need for broader validation methods ⚖️ The guidance is currently advisory but aims to set a higher standard for AI device approvals while addressing public trust in AI technologies 👇Link to articles and draft guidance in comments #digitalhealth #FDA #AI

🚨 Why Enterprise AI Doesn’t Fail Because of Bad Models: It Fails Because of Broken Trust Most AI teams build features first and try to earn trust later. We flipped that model. At Calonji Inc., we built MedAlly.ai, a multilingual, HIPAA-compliant GenAI platform, by starting with what matters most in enterprise AI: ✅ Trust. Not as a UI layer. Not as a compliance checklist. ✅ But as the core architecture. Here’s the Trust Stack that changed everything for us: 🔍 Explainability = Adoption 📡 Observability = Confidence 🚧 Guardrails = Safety 📝 Accountability = Defensibility This wasn’t theory. It drove real business outcomes: ✔️ 32% increase in user adoption ✔️ Faster procurement and legal approvals ✔️ No undetected model drift in production 📌 If your platform can't answer "why," show behavior transparently, or survive a trust audit, it's not ready for enterprise scale. Let’s talk: What’s in your Trust Stack? #EnterpriseAI #AITrust #ExplainableAI #AIArchitecture #ResponsibleAI #SaaS #CTOInsights #PlatformStrategy #HealthcareAI #DigitalTransformation