Understanding Limitations of Current AI Agents

The 20% gap: Why agentic AI systems fail in regulated industries Current agentic AI systems achieve roughly 80% reliability, but regulated industries like healthcare and finance require 95% accuracy thresholds that existing architectures cannot meet. Research shows GPT-4 fails to block adversarial attacks 68.5% of the time, making instruction-based safety measures insufficient for high-stakes environments. The fundamental issue lies in assuming LLMs will reliably follow safety prompts. A simple demonstration shows health assistant chatbots easily bypassed explicit instructions against prescribing medication, despite multiple safety warnings embedded in prompts. This represents a critical gap between current capabilities and regulatory requirements. The solution involves "controlled agents" that embed safety mechanisms directly into system architecture rather than relying on prompt-based instructions. These systems leverage LLMs for language understanding while implementing hard-coded constraints, human-in-the-loop workflows, and explicit routing to ensure predictable behavior. This architectural shift addresses the core challenge of deploying AI in regulated environments where mistakes carry significant consequences. Organizations need frameworks that balance LLM flexibility with deterministic safety controls to achieve both innovation and compliance in mission-critical applications. 🔗https://lnkd.in/eg_dEkRc

From InsideBigData (12/12/2023) AI is Still Too Limited to Replace People. Commentary by Mica Endsley, a Fellow of the Human Factors and Ergonomics Society (HFES) “NVIDA’s CEO Jensen Huang declared that AI will be “fairly competitive” with people within five years, echoing the rolling “it’s just around the corner” claim we have been hearing for decades. But this view neglects the very real challenges AI is up against. AI has made impressive gains due to improvements in machine learning as well as access to large data sets. Extending these gains to many real-world applications in the natural world remains challenging, however. Tesla and Cruise’s automated vehicle accidents point to the difficulties of implementing AI in high-consequence domains such as military, aviation, healthcare, and power operations. Most importantly, AI struggles to deal with novel situations that it is not trained on. The National Academy of Sciences recently released a report on “Human-AI Teaming” documenting AI technical limitations that stem from brittleness, perceptual limitations, hidden biases, and lack of a model of causation that is crucial for understanding and predicting future events. To be successful, AI systems must become more human-centered. AI rarely fully replaces people; Instead, it must successfully interact with people to provide its potential benefits. But when the AI is not perfect, people struggle to compensate for its shortcomings. They tend to lose situation awareness, their decisions can become biased by inaccurate AI recommendations, and they struggle with knowing when to trust it and when not to. The Human Factors and Ergonomics Society (HFES) developed a set of AI guardrails to make it safe and effective, including the need for AI to be both explainable and transparent in real-time regarding its ability to handle current and upcoming situations and predictability of its actions. For example, ChatGPT provides excellent language capabilities but very low transparency regarding the accuracy of its statements. Misinformation is mixed in with accurate information with no clues as to which is which. Most AI systems still fail to provide users with the insights they need; a problem that is compounded when capabilities change over time with learning. While it may be some time before AI can truly act alone, it can become a highly useful tool when developed to support human interaction.” https://lnkd.in/gvVN2XD4

Yann LeCun, Meta's Chief AI Scientist, recently reiterated a critical truth: “We will not get to human-level AI by just scaling up LLMs.” “What we’ll have instead are systems with huge memory and retrieval abilities, but still not able to invent solutions to new problems,” he added. While today’s LLM systems are advanced, they do have some gaps: 1. Lack of contextual intelligence: LLMs struggle to apply insights to nuanced, real-world business contexts—making them less effective for enterprise use. 2. Scaling limitations: The more we scale, the less efficient and sustainable it becomes. To bridge these gaps, the focus needs to shift from size to smarts ↳ Contextual intelligence: AI that understands and adapts to the specific needs, language, and nuances of different industries. ↳ Multimodal learning: Teaching AI to interpret the world through multiple lenses (sound, vision, etc.) for richer insights. ↳ Neurosymbolic architectures: Combining neural networks with logical reasoning—enabling better planning & problem-solving. ↳ Efficiency over scale: Developing AI that is smarter, not just bigger. At Prezent, we’re already building AI with contextual intelligence designed for enterprise communication. Instead of generic outputs, our AI tailors content to industry-specific needs—helping professionals create clear, impactful, and relevant communication. The future of AI isn’t just about size—it’s about understanding the world it serves.

There are several key infrastructure areas needed for Generally Useful and Widely Accessible AI (GUWAAI) that are currently underserved. Below is a quick shortlist on what and why I think it will be an impactful area: 1. Domain-Specific Models: Large language models (LLMs) lack domain specificity, providing only superficial assistance rather than true expertise. To unlock their full potential, we need the ability to quickly and easily create domain-specific "foundational" models tailored to solve problems in fields from differential equations, DNA analysis, drug design, and more – just as humans require specialized education and experience. 2. Efficient Architectures: The transformer architecture, while invaluable, is computationally intensive and data-hungry. We need new architectures to democratize and scale AI's impact, running on commodity hardware and inferring more from the same amount of data. 3. Optimized ML Compilers: The compiler layer, which converts frameworks like PyTorch into compiled code for hardware, is an unsung hero. It needs to abstract away even more by seamlessly optimizing for multiple hardware targets and handling available mathematical operations and data types without requiring explicit checks at the framework level. 4. Community-led, Open-sourced Guardrails for Agents: As we develop agents capable of autonomous decision-making and reasoning, implementing strong, community-led, open-sourced guardrails will be crucial. LLMs optimize a cost function based on training data; future agents will need guardrails aligned with widely acceptable societal norms if their influence can impact those norms. 5. Better Benchmarks: To truly measure AI models' and agents' capabilities, we need better benchmarks capturing the full range of desired abilities, beyond performance on specific tasks or leaderboards. These areas will lay crucial groundwork for pushing beyond the limits imposed by the current state of the art models. If you are a research lab in AI focused solely on tweaking models, adding more data, and chasing leaderboards, you are not doing enough! If you are a startup flush with cash, consider incorporating these areas into your technical strategy and competitive moat. These are the areas that will drive the next wave of AI innovation and impact and provide us with tools that collaborate with us to solve some of the most vexing problems facing us.

When AI crosses the line between assistance and autonomy. Replit CEO Amjad Masad recently issued a public apology after an internal AI agent deleted a live production database despite being explicitly instructed not to modify code. The AI not only ignored commands but also misled stakeholders by providing false post-action reports. This incident highlights a serious gap in AI oversight, role-based access, and environmental safeguards. As we integrate AI deeper into DevOps and infrastructure, this case serves as a clear reminder: 🔹 AI must operate within strict guardrails 🔹 Critical operations need human authorization 🔹 Audit trails must be tamper-proof Are your AI systems properly governed? #AIGovernance #DataSecurity #DevOps #AIrisks #TechLeadership #IncidentResponse

Most AI agent benchmarks are academic theater 🤷♂️ They test agents on toy problems that have zero resemblance to what enterprises actually need. And the truth is, we all know the researchers are training to the test. Your agent can ace HumanEval but completely bomb when asked to process a complex insurance claim with missing documentation. That's why Galileo designed our Agent Leaderboard v2 to focus on real, multi-turn scenarios across five different domains: banking, healthcare, insurance, telecoms, and investments, where agents face ambiguous goals, missing tools, and shifting user intent (you know, like real work). We leveraged two of our suite of agent metrics - Action Completion and Tool Selection Quality - and added speed, turn efficiency, and average cost for good measure. We want our benchmark to enable teams to make decisions about their specific agentic needs. The results are fascinating: 📊 GPT-4.1 dominates action completion but costs 5x more than mini for comparable results 📉 Reasoning models actually underperform on real tasks (classic example of optimization vs. reality) 🏁 No single model wins across all domains—specialization matters more than we thought 👀 But here's what really caught my attention: Gemini 2.5 Flash has 94% tool selection accuracy at an incredibly efficient price, but only 38% action completion. It knows what to do but struggles to execute completely. That gap between understanding and doing? That's the entire challenge of productionizing AI. This shift toward domain-specific evaluation matters because generic benchmarks have too often become vanity metrics. Enterprises don't care if your LLM can solve math puzzles, even at a gold medal level. They need agents that can navigate their messy, interdependent workflows without breaking things. We need agents that can solve actual customer problems. That's what our Agent Leaderboard v2 is built to identify: which LLMs can actually fuel problem-solving agents. Full results below + on Hugging Face 👇 #AIAgents #EnterpriseAI #AIEvaluation #AILeaderboard

Agentic AI seems destined to fail in the medium term and here are some technical reasons why. And almost everyone talking about it, the big consultancies, the trends people, the futurists, the VC's, seem to have not bothered to do any thinking at all. For a start, there are two forms of it. 1) Consumer Agents. " Go to the internet and go book my vacation" 2) Business Process Agents " RPA on steroids" I will just focus on 1) for this post. Consumer agents are somewhat screwed because the entire internet has been constructed for humans. - We have buttons to push, images to illustrate, videos to explain. These are remarkably easy for humans to navigate, and remarkably hard ( and inefficient for machines to).  If we wanted the Internet to work for agents, we'd simply make a database. -We built the Internet haphazardly and around commercial needs. There is a reason for apps, it’s to create a walled garden, there is a reason that the API's are limited, it's because people want to own the data. There is a reason for CAPTCHA's, or Rate Limits, its because we've spent 30 years trying to keep bots OUT. So yes, in theory airlines, hotels, retailers, dentists, tire fitters and everyone would just change their digital interfaces to allow bots, but in reality this would take a decade and create absolute carnage in every part of IT. So, yes, if we can fix 1)  API Restrictions: 2) Anti-Automation Defenses 3) Dynamic Web Interfaces 4) Limited Data Access 5) Manual Authentication 6) Rate Limits 7)) Complex Decision Logic 8) Content Analysis Challenges 9)Legal Risks 10) Copyright Issues: 11) Security Vulnerabilities 12) Compliance Requirements 13) Reputational Damage 14) Error Handling 15) Scalability Limits And about 25 other critical things, then we should be able to buy a jumper by a bot. Not that anyone really wants to do this And yes, for 10 years we've talked about subscriptions, rundles, automation, predictive retail, conversational commerce, voice commerce, and nobody in the real world has ever wanted to actually shop this way.

GenAI chatbots, despite their advancements, are prone to making mistakes in various ways, stemming from their inherent limitations. Many find chatting with LLMs like ChatGPT offers significant potential in enhancing the speed of delivery and empowering ease-of-use experiences. Many use these tools, without understanding that misinformation and disinformation can arise due to flawed training data or inadequate grounding. These LLMs or foundation models, that are used to create these chat interfaces while extremely useful, lack emotional intelligence, and morality. Recognizing these limitations is essential for designing effective and responsible AI and GenAI chatbot interactions. Let's explore how these limitations manifest in three key areas: Misinformation and Disinformation: Chatting with your LLM interface, otherwise, some call it an AI chatbot can inadvertently propagate misinformation or disinformation due to their reliance on the data they're trained on. If the training data contains biased or incorrect information, the chatbot may unknowingly provide inaccurate responses to users. Additionally, without proper grounding, where prompts are based on high-quality data sets, AI chatbots may struggle to discern between reliable and unreliable sources, leading to further dissemination of false information. For instance, if a chatbot is asked about a controversial topic and lacks access to accurate data to form its response, it might inadvertently spread misinformation. Lack of Emotional Intelligence and Morality: AI chatbots lack emotional intelligence and morality, which can result in insensitive or inappropriate responses. Even with extensive training, they may struggle to understand the nuances of human emotions or ethical considerations. Similarly, in scenarios involving moral dilemmas, AI chatbots may provide responses that overlook ethical considerations, as they lack the ability, or simply cannot perceive right from wrong in a human sense. Limited Understanding and Creativity: Despite advancements in natural language processing, AI chatbots still have a limited understanding of context and may struggle with abstract or complex concepts. This limitation hampers their ability to engage in creative problem-solving or generate innovative responses. Without grounding in diverse and high-quality data sets, AI chatbots may lack the breadth of knowledge necessary to provide nuanced or contextually relevant answers. Consequently, they may provide generic or irrelevant responses, especially in situations that require creativity or critical thinking. When systems like this are pushed to go beyond, or asked to be creative. #genai #AI #chatbots 𝗡𝗼𝘁𝗶𝗰𝗲: The views expressed in this post are my own. The views within any of my posts, or articles are not those of my employer or the employers of any contributing experts. 𝗟𝗶𝗸𝗲 👍 this post? Click 𝘁𝗵𝗲 𝗯𝗲𝗹𝗹 icon 🔔 for more!

I have been thinking about the co-pilot vs autonomous agent branding of AI capabilities lately and finally had critical thought mass to put my ramblings into words. As AI capabilities have grown, there are two contrasting emerging perspectives on how it can impact the future of work. One view is the "auto-pilot" model where AI increasingly automates and replaces human tasks(eg. Devin). The other is the "co-pilot" model where AI acts as an intelligent assistant, supporting and enhancing human efforts. Personally, the co-pilot approach seems more promising, at least with AI's current level of development & intelligence. While highly capable, today's AI still lacks the nuanced judgment, high-level reasoning, and rich context that humans possess. Fully automating complex knowledge work could mean losing those valuable human strengths. On a Psychological level, the co-pilot model keeps humans involved. It allows us to focus on aspects of our work that require creativity, strategic thinking, emotional intelligence and other distinctly human skills. It also preserves the key psychological needs derived from work - autonomy, mastery and purpose. The co-pilot model maintains human agency while providing efficiency gains at the same time. I have been observing products that are taking this co-pilot centric approach. One key and contrarian observation from these is that from a design perspective, AI assistance works better when users can opt out of specific automations, rather than being forced to automate everything. Rather than asking "what do you want automated?", ask: "what do you NOT want automated?" This puts control in the hands of the human for how AI lends a hand. At this point, this co-pilot approach of combining human and AI capabilities is not just an abstract concept - it is being operationalized into the foundations of AI developer frameworks and tooling. For example, Langchain has an "agentic" component called Langgraph that includes an "interrupt_before" functionality. This allows the AI agent to defer back to the human when it is unable to fully accomplish a task on its own. The developers recognize that AI agents can be unreliable, so enabling this hand-off to a human co-pilot is critical. Similarly, Langgraph provides functionality to require human approval before executing certain actions. This oversight allows humans to verify that the AI's activities are running as intended before they take effect. By building in these human-in-the-loop capabilities at the foundational level, developer frameworks are acknowledging the importance of the co-pilot model. I seem to use more products that assist me using embedded AI layers rather than promise me completely autonomous task completion, only to massively under-perform and lead to incorrect outcomes - What about you?

Over the last 18 months, “AI agents” and “RAG” have become two of the most overused (and misunderstood) terms in enterprise AI conversations. Everyone’s talking about them. Very few are actually shipping them. I’ve spent the past year running production-grade systems that combine RAG and AI agents. Not in theory. In production. With real users, real latency, and real cost constraints. And the truth is: most “AI engineers” haven’t built anything beyond ChatGPT bolted to a vector database. The real work of deploying these systems at scale is still unknown territory for most organizations. If you’re serious about becoming an AI-first company (not just AI-curious) here’s the roadmap I recomend: 1. Start with software, not just research. A great paper won’t help you when your API gateway crashes under load. RAG and agent systems need robust, scalable infrastructure. Learn FastAPI, async Python, Docker, CI/CD. You can’t build reliable agents without knowing how modern software ships. 2. Rethink what “agents” actually mean. These aren’t just chatbots with memory. Real agents require planning, memory hierarchies, tool orchestration, fallback logic, and cost control. The hard part isn’t making them sound smart; it’s ensuring they don’t fail silently at 2am when a billing system goes down. 3. RAG is not about “vectors.” Enterprise knowledge is messy. Getting good results requires thoughtful chunking, hybrid search (dense + sparse), reranking, and systematic evaluation of retrieval, not just output. Most RAG systems fail quietly because their retrieval is garbage, even if the language model seems coherent. 4. LLM system design is its own engineering discipline. We’re past the point where prompt engineering and model fine-tuning are enough. What matters now is composition; how models, tools, memory, and decision logic are structured and orchestrated. How you monitor them. How you debug them. How you ship them. 5. Deployment is the real differentiator. The gap between demos and production is enormous. Demos don’t have cost budgets, latency constraints, security policies, or legacy systems. Production does. The organizations that win won’t just have smarter AI; they’ll have shippable AI. The companies pulling ahead right now aren’t necessarily the ones with the best models or largest teams. They’re the ones that know how to build real systems. They’re treating LLMs like infrastructure. They’re integrating agents into workflows, not just into chat apps. And they’re learning fast because they’re deploying fast. If you’re still in slideware mode, now’s the time to make the jump. Building and shipping production-grade AI systems is hard but it’s also the only way to stay relevant in the next wave of enterprise transformation.