Building AI-Powered Recommendation Systems

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

𝗔𝗜 𝗠𝗼𝗱𝗲𝗹𝘀: 𝗧𝗵𝗲 𝗚𝗼𝗼𝗱, 𝗧𝗵𝗲 𝗕𝗮𝗱, 𝗮𝗻𝗱 𝗧𝗵𝗲... 𝗕𝗶𝗮𝘀𝗲𝗱? 🤔 Today we will discuss how process of building a model based on data can introduce a whole new set of biases into the AI funnel. Yesterday we focused the aspect of data biases And how those biases can trickle down to the next level of the AI funnel unless they are identified and corrected. The purpose of building a model is to create a predictive output based on the expected input from a user with the help of available data. It is often referred to as the "𝗯𝗹𝗮𝗰𝗸 𝗯𝗼𝘅" of the AI tool because the components of models are often too technical and complex for most folks to delve into. Data and development teams are usually the most informed about the models and codes that were uses in the programming. In the pursuit of fancy AI models, sometimes the simplest and most relevant models get overlooked. This is why it is critical to ensure that models are being built under the guidance of a statistical expert. Many of these biases are referred to collectively, as "Algorithmic Biases" in articles, reports, and other sources of information. 𝗦𝗼𝗺𝗲 𝗖𝗼𝗺𝗺𝗼𝗻 𝗠𝗼𝗱𝗲𝗹-𝗥𝗲𝗹𝗮𝘁𝗲𝗱 𝗕𝗶𝗮𝘀𝗲𝘀 1. Model Selection Bias: - Different models have different strengths and weaknesses - Selecting the wrong model can lead to biased outcomes. 2. Feature Selection Bias: - Omitting relevant features (variables) from model - Including irrelevant features from model - In good ol' statistics, we call these "omitted variable bias" and "precision bias," respectively. 3. Assumption Bias: - Each mathematical model is based upon certain assumptions. - These assumptions may be violated by the data distribution - Incorrect assumptions can lead to biased outcomes. 4. Training and Testing Bias: - Training data may be clustered or biased - Objectivity and reliability need to be correctly tested - Training and testing data should represent reality. 5. Deployment Bias: - If a bad model gets deployed without addressing these issues, the initial failure of the model in getting expected results can affect future reliability - Users will be more reluctant to adopt a corrected model because of the reputational damage. Model diversity, reliability and objectivity matter. Even if data was unbiased, model selection or deployment can inject bias into the AI funnel. Tomorrow, we will explore the biases that occur at the User end of the AI funnel. 👉 Don’t underestimate the impacts of biases at each stage of this funnel. #PostItStatistics #DataScience #ai LinkedIn Got questions? Drop a comment ⬇ and I will do my best to clarify any confusions! ******************************************************** 🔔 Follow me or Analytics TX, LLC to see more nuggets like these. ✍ DM me to simplify your complex data and increase your top or bottom lines!

You've built your AI agent... but how do you know it's not failing silently in production? Building AI agents is only the beginning. If you’re thinking of shipping agents into production without a solid evaluation loop, you’re setting yourself up for silent failures, wasted compute, and eventully broken trust. Here’s how to make your AI agents production-ready with a clear, actionable evaluation framework: 𝟭. 𝗜𝗻𝘀𝘁𝗿𝘂𝗺𝗲𝗻𝘁 𝘁𝗵𝗲 𝗥𝗼𝘂𝘁𝗲𝗿 The router is your agent’s control center. Make sure you’re logging: - Function Selection: Which skill or tool did it choose? Was it the right one for the input? - Parameter Extraction: Did it extract the correct arguments? Were they formatted and passed correctly? ✅ Action: Add logs and traces to every routing decision. Measure correctness on real queries, not just happy paths. 𝟮. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿 𝘁𝗵𝗲 𝗦𝗸𝗶𝗹𝗹𝘀 These are your execution blocks; API calls, RAG pipelines, code snippets, etc. You need to track: - Task Execution: Did the function run successfully? - Output Validity: Was the result accurate, complete, and usable? ✅ Action: Wrap skills with validation checks. Add fallback logic if a skill returns an invalid or incomplete response. 𝟯. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝘁𝗵𝗲 𝗣𝗮𝘁𝗵 This is where most agents break down in production: taking too many steps or producing inconsistent outcomes. Track: - Step Count: How many hops did it take to get to a result? - Behavior Consistency: Does the agent respond the same way to similar inputs? ✅ Action: Set thresholds for max steps per query. Create dashboards to visualize behavior drift over time. 𝟰. 𝗗𝗲𝗳𝗶𝗻𝗲 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗧𝗵𝗮𝘁 𝗠𝗮𝘁𝘁𝗲𝗿 Don’t just measure token count or latency. Tie success to outcomes. Examples: - Was the support ticket resolved? - Did the agent generate correct code? - Was the user satisfied? ✅ Action: Align evaluation metrics with real business KPIs. Share them with product and ops teams. Make it measurable. Make it observable. Make it reliable. That’s how enterprises scale AI agents. Easier said than done.

If you want to know where the money is in Machine Learning, look no further than Recommender Systems! Recommender systems are usually a set of Machine Learning models that rank items and recommend them to users. We tend to care primarily about the top-ranked items, the rest being less critical. If we want to assess the quality of a specific recommendation, typical ML metrics may be less relevant. Let’s take the search results of a Google search query, for example. All the results are somewhat relevant, but we need to make sure that the most relevant items are at the top of the list. To capture the level of relevance, it is common to hire human labelers to rate the search results. It is a very expensive process and can be quite subjective since it involves humans. For example, we know that Google performed 757,583 search quality tests in 2021 using human raters: https://lnkd.in/gYqmmT2S. Normalized Discounted Cumulative Gain (NDCG) is a common metric to exploit relevance measured on a continuous spectrum. Let’s break that metric down. Using the relevance labels we can compute diverse metrics to measure the quality of the recommendation. The cumulative gain (CG) metric answers the question: How much relevance is contained in the recommended list? To get a quantitative answer to that question, we simply add the relevance scores provided by the labeler: CG = relevance 1 + relevance 2 + ... The problem with cumulative gain is that it doesn’t take into account the position of the search results. Any order would give the same value however we want the most relevant items at the top. Discounted cumulative gain (DCG) discounts relevance scores based on their position in the list. The discount is usually done with a log function, but other monotonic functions could be used: DCG = relevance 1 / log(position 1) + relevance 2 / log(position 2) + ... DCG is quite dependent on the specific values used to describe relevance. Even with strict guidelines, some labelers may use high numbers and others low numbers. To put those different DCG values on the same level, we normalize them by the highest value DCG can take. The highest value corresponds to the ideal ordering of the recommended items. We call the DCG for ideal ordering the Ideal Discounted Cumulative Gain (IDCG). The Normalized Discounted Cumulative Gain (NDCG) is the normalized DCG NDCG = DCG / IDCG If the relevance scores are all positive, then NDCG is contained in the range [0, 1], where 1 is the ideal ordering of the recommendation. #MachineLearning #DataScience #ArtificialIntelligence

If you are an AI Engineer building production-grade GenAI systems, RAG should be in your toolkit. LLMs are powerful for information generation, but: → They hallucinate → They don’t know anything post-training → They struggle with out-of-distribution queries RAG solves this by injecting external knowledge at inference time. But basic RAG (retrieval + generation) isn’t enough for complex use cases. You need advanced techniques to make it reliable in production. Let’s break it down 👇 🧠 Basic RAG = Retrieval → Generation You ask a question. → The retriever fetches top-k documents (via vector search, BM25, etc.) → The LLM answers based on the query + retrieved context But, this naive setup fails quickly in the wild. You need to address two hard problems: 1. Are we retrieving the right documents? 2. Is the generator actually using them faithfully? ⚙️ Advanced RAG = Engineering Both Ends To improve retrieval, we have techniques like: → Chunk size tuning (fixed vs. recursive splitting) → Sliding window chunking (for dense docs) → Structured data retrieval (tables, graphs, SQL) → Metadata-aware search (filtering by author/date/type) → Mixed retrieval (hybrid keyword + dense) → Embedding fine-tuning (aligning to domain-specific semantics) → Question rewriting (to improve recall) To improve generation, options include: → Compressing retrieved docs (summarization, reranking) → Generator fine-tuning (rewarding citation usage and reasoning) → Re-ranking outputs (scoring factuality or domain accuracy) → Plug-and-play adapters (LoRA, QLoRA, etc.) 🧪 Beyond Modular: Joint Optimization Some of the most promising work goes further: → Fine-tuning retriever + generator end-to-end → Retrieval training via generation loss (REACT, RETRO-style) → Generator-enhanced search (LLM reformulates the query for better retrieval) This is where RAG starts to feel less like a bolt-on patch and more like a full-stack system. 📏 How Do You Know It's Working? Key metrics to track: → Context Relevance (Are the right docs retrieved?) → Answer Faithfulness (Did the LLM stay grounded?) → Negative Rejection (Does it avoid answering when nothing relevant is retrieved?) → Tools: RAGAS, FaithfulQA, nDCG, Recall@k 🛠️ Arvind and I are kicking off a hands-on workshop on RAG This first session is designed for beginner to intermediate practitioners who want to move beyond theory and actually build. Here’s what you’ll learn: → How RAG enhances LLMs with real-time, contextual data → Core concepts: vector DBs, indexing, reranking, fusion → Build a working RAG pipeline using LangChain + Pinecone → Explore no-code/low-code setups and real-world use cases If you're serious about building with LLMs, this is where you start. 📅 Save your seat and join us live: https://lnkd.in/gS_B7_7d Image source: LlamaIndex

Want to Make Your RAG Application 10x Smarter? Retrieval-Augmented Generation (RAG) systems are powerful, however with the right strategies, you can turn them into precision tools. Here’s a breakdown of 10 expert-backed ways to optimize RAG performance: 1. 🔹Use Domain-Specific Embeddings Choose embeddings trained on your industry (like legal, medical, or finance) to improve semantic understanding and relevance. 2. 🔹Chunk Wisely Split documents into overlapping, context-rich chunks. Avoid mid-sentence breaks to preserve meaning during retrieval. 3. 🔹Rerank Results with LLMs Instead of relying only on top vector matches, rerank retrieved chunks using your LLM and a scoring prompt. 4. 🔹Add Metadata Filtering Use filters (like author, date, or doc type) to refine results before sending them to your language model. 5. 🔹Use Hybrid Search (Vector + Keyword) Combine the precision of keyword search with the flexibility of vector search to boost accuracy and recall. [Explore More In The Post] ✅ Use this checklist to fine-tune your RAG workflows, reduce errors, and deliver smarter, more reliable AI responses. #genai #artificialintelligence

RAG just got smarter. If you’ve been working with Retrieval-Augmented Generation (RAG), you probably know the basic setup: An LLM retrieves documents based on a query and uses them to generate better, grounded responses. But as use cases get more complex, we need more advanced retrieval strategies—and that’s where these four techniques come in: Self-Query Retriever Instead of relying on static prompts, the model creates its own structured query based on metadata. Let’s say a user asks: “What are the reviews with a score greater than 7 that say bad things about the movie?” This technique breaks that down into query + filter logic, letting the model interact directly with structured data (like Chroma DB) using the right filters. Parent Document Retriever Here, retrieval happens in two stages: 1. Identify the most relevant chunks 2. Pull in their parent documents for full context This ensures you don’t lose meaning just because information was split across small segments. Contextual Compression Retriever (Reranker) Sometimes the top retrieved documents are… close, but not quite right. This reranker pulls the top K (say 4) documents, then uses a transformer + reranker (like Cohere) to compress and re-rank the results based on both query and context—keeping only the most relevant bits. Multi-Vector Retrieval Architecture Instead of matching a single vector per document, this method breaks both queries and documents into multiple token-level vectors using models like ColBERT. The retrieval happens across all vectors—giving you higher recall and more precise results for dense, knowledge-rich tasks. These aren’t just fancy tricks. They solve real-world problems like: • “My agent’s answer missed part of the doc.” • “Why is the model returning irrelevant data?” • “How can I ground this LLM more effectively in enterprise knowledge?” As RAG continues to scale, these kinds of techniques are becoming foundational. So if you’re building search-heavy or knowledge-aware AI systems, it’s time to level up beyond basic retrieval. Which of these approaches are you most excited to experiment with? #ai #agents #rag #theravitshow

Data flywheels accelerate AI product development and create competitive advantages along the way. Here’s the strategy NVIDIA and Microsoft use to deliver highly reliable AI products faster. Data flywheels are a critical component of AI product design that take advantage of a unique property of AI platforms. Contextual data improves models, and every time a user does work on a platform, they generate data with the context of a workflow. A data flywheel is a self-reinforcing cycle where the collection and analysis of data lead to continuous improvements in products or services, attracting more users who generate additional data, which perpetuates the cycle or flywheel. Here’s how it works in practice ▶️ 1️⃣ Identify a Specific Problem: Focus on a workflow that data and models can support in a way that current technical solutions don’t. 2️⃣ Gather Contextual Data: Engineer access to the workflow to gather data in the context of tasks, decisions, and outcomes. 3️⃣ Analyze the Data: Extract actionable insights about the workflow from the contextual data and identify opportunities for improvement that deliver new value to customers or users. 4️⃣ Implement Improvements: Use the contextual data to introduce analytics to the workflow and train reliable models that improve how the AI product supports the workflow. 5️⃣ Generate More Contextual Data: As improvements are implemented, they lead to increased usage or engagement, resulting in the collection of more data, which feeds back into the cycle. Netflix's recommendation system improved through a data flywheel. Initially, Netflix recommended the same popular videos to all users. Analyzing individual viewing habits allowed Netflix to retrain its models to offer more personalized suggestions. Personalization prevented churn and increased the time people spent watching Netflix, generating additional data that further improved the recommendation models’ accuracy, creating a virtuous cycle of improvement. Data flywheels lead to arena learning or learning via simulations. Both Microsoft and NVIDIA have shown the power of this paradigm. Expect more companies to follow their lead.

Why would your users distrust flawless systems? Recent data shows 40% of leaders identify explainability as a major GenAI adoption risk, yet only 17% are actually addressing it. This gap determines whether humans accept or override AI-driven insights. As founders building AI-powered solutions, we face a counterintuitive truth: technically superior models often deliver worse business outcomes because skeptical users simply ignore them. The most successful implementations reveal that interpretability isn't about exposing mathematical gradients—it's about delivering stakeholder-specific narratives that build confidence. Three practical strategies separate winning AI products from those gathering dust: 1️⃣ Progressive disclosure layers Different stakeholders need different explanations. Your dashboard should let users drill from plain-language assessments to increasingly technical evidence. 2️⃣ Simulatability tests Can your users predict what your system will do next in familiar scenarios? When users can anticipate AI behavior with >80% accuracy, trust metrics improve dramatically. Run regular "prediction exercises" with early users to identify where your system's logic feels alien. 3️⃣ Auditable memory systems Every autonomous step should log its chain-of-thought in domain language. These records serve multiple purposes: incident investigation, training data, and regulatory compliance. They become invaluable when problems occur, providing immediate visibility into decision paths. For early-stage companies, these trust-building mechanisms are more than luxuries. They accelerate adoption. When selling to enterprises or regulated industries, they're table stakes. The fastest-growing AI companies don't just build better algorithms - they build better trust interfaces. While resources may be constrained, embedding these principles early costs far less than retrofitting them after hitting an adoption ceiling. Small teams can implement "minimum viable trust" versions of these strategies with focused effort. Building AI products is fundamentally about creating trust interfaces, not just algorithmic performance. #startups #founders #growth #ai