How to Integrate Feedback Loops into Workflows

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

Critique this (real) team's experiment. Good? Bad? Caveats? Gotchas? Contexts where it will not work? Read on: Overview The team has observed that devs often encounter friction during their work—tooling, debt, environment, etc. These issues (while manageable) tend to slow down progress and are often recurring. Historically, recording, prioritizing, and getting approval to address these areas of friction involves too much overhead, which 1) makes the team less productive, and 2) results in the issues remaining unresolved. For various reasons, team members don't currently feel empowered to address these issues as part of their normal work. Purpose Empower devs to address friction points as they encounter them, w/o needing to get permission, provided the issue can be resolved in 3d or less. Hypothesis: by immediately tackling these problems, the team will improve overall productivity and make work more enjoyable. Reinforce the practice of addressing friction as part of the developers' workflow, helping to build muscle memory and normalize "fix as you go." Key Guidelines 1. When a dev encounters friction, assess whether the issue is likely to recur and affect others. If they believe it can be resolved in 3d or less, they create a "friction workdown" ticket in Jira (use the right tags). No permission needed. 2. Put current work in "paused" status, mark new ticket as "in progress," and notify the team via #friction Slack channel with a link to the ticket. 3. If the dev finds that the issue will take longer than 3d to resolve, they stop, document what they’ve learned, and pause the ticket. This allows the team to revisit the issue later and consider more comprehensive solutions. This is OK! 4. After every 10 friction workdown tickets are completed, the team holds a review session to discuss the decisions made and the impact of the work. Promote transparency and alignment on the value of the issues addressed. 5. Expires after 3mos. If the team sees evidence of improved efficiency and productivity, they may choose to continue; otherwise, it will be discontinued (default to discontinue, to avoid Zombie Process). 6. IMPORTANT: The team will not be asked to cut corners elsewhere (or work harder) to make arbitrary deadlines due to this work. This is considered real work. Expected Outcomes Reduce overhead associated with addressing recurring friction points, empowering developers to act when issues are most salient (and they are motivated). Impact will be measured through existing DX survey, lead time, and cycle time metrics, etc. Signs of Concern (Monitor for these and dampen) 1. Consistently underestimating the time required to address friction issues, leading to frequent pauses and unfinished work. 2. Feedback indicating that the friction points being addressed are not significantly benefiting the team as a whole. Limitations Not intended to impact more complex, systemic issues or challenges that extend beyond the team's scope of influence.

The #1 reason people don't use AI in their workflows (and how to fix it) In a recent Supra Insider podcast, Jacob Bank from Relay.app shared a powerful playbook for effective AI implementation. His critical insight: "The main reason people don't use AI in practice right now is not because they haven't heard of it, not because they don't think it's cool... just because they can't trust it to do work on their behalves." The solution? Human-in-the-loop design. Instead of viewing AI as "fully automated or not," successful implementations create thoughtful checkpoints where humans remain in control: 1/ Plan transparency Before executing, AI should communicate its approach to the task. This creates confidence by letting users understand what will happen. Without this step, users fear uncontrolled actions like "writing 5,000 emails to every customer individually" or running up costs unnecessarily. Examples: "Here's how I'll tackle this task and where I'll need your input." 2/ Refinement opportunities Create explicit moments where humans can guide the AI's work while it's in progress. These aren't just approval checkpoints but collaborative interactions. These refinement stages are perfect for content creation, telling the AI to "emphasize this part of the conversation more, this part less, go back and try again." Examples: ↳ "This looks good, but emphasize this part more" ↳ "These results need context from last quarter" ↳ "You're missing an important constraint" 3/ Quality assurance gates Establish critical approval points that cannot be bypassed before final output. For successful AI workflows like LinkedIn content creation, never let AI publish directly. For important workflows, multiple QA checkpoints are essential - first reviewing the draft, then refining for polish, and finally a human edit before publishing. Examples: ↳ "Review this draft before sending" ↳ "Confirm these metrics are accurate" ↳ "Approve this selection of priority items" 4/ Outcome verification Close the loop by providing feedback on results to improve future performance. This step makes AI tools progressively more valuable over time. Use this approach to refine content workflows by analyzing which posts perform well and feeding that data back into the system. Examples: ↳ "The approach worked, but next time include X" ↳ "This missed the mark because of Y" ↳ "This exceeded expectations, let's rely on it more" Even with perfect prompts, AI drafts typically only get "80% of the way to the quality bar" needed for publication. The companies winning with AI aren't eliminating humans from the process. They're creating thoughtful collaboration points that leverage the strengths of both. Where are you implementing human-in-the-loop design in your AI workflows? What checkpoints have you found most valuable?

𝗜𝗳 𝗬𝗼𝘂’𝗿𝗲 𝗡𝗼𝘁 𝗗𝗲𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗟𝗼𝗼𝗽𝘀, 𝗬𝗼𝘂’𝗿𝗲 𝗡𝗼𝘁 𝗗𝗲𝘀𝗶𝗴𝗻𝗶𝗻𝗴 𝗦𝘆𝘀𝘁𝗲𝗺𝘀, 𝗬𝗼𝘂’𝗿𝗲 𝗚𝘂𝗲𝘀𝘀𝗶𝗻𝗴. In service design and journey management, we talk a lot about touchpoints, channels, and experiences. 𝗛𝗲𝗿𝗲’𝘀 𝘁𝗵𝗲 𝘁𝗿𝘂𝘁𝗵: - No journey gets better without feedback. - No system evolves without learning loops. A feedback loop is the engine that turns friction into insight, and insight into action. In great systems, feedback loops are: 1. 𝗩𝗶𝘀𝗶𝗯𝗹𝗲 – Customers, brokers, employees can see the impact of their feedback 2. 𝗧𝗶𝗺𝗲𝗹𝘆 – Data isn’t stuck in a quarterly report, it’s now 3. 𝗔𝗰𝘁𝗶𝗼𝗻𝗮𝗯𝗹𝗲 – It doesn’t just inform, it drives change 4. 𝗖𝗹𝗼𝘀𝗲𝗱 – People know they’ve been heard 𝗜𝗻 𝗯𝗿𝗼𝗸𝗲𝗻 𝘀𝘆𝘀𝘁𝗲𝗺𝘀, 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗱𝗶𝗲𝘀 𝗶𝗻:  🚫 Static maps and surveys nobody reads  🚫 Call logs without analysis  🚫 Dashboards with no ownership  🚫 “That’s just how the process works” 𝗧𝗵𝗶𝗻𝗸 𝗮𝗯𝗼𝘂𝘁 𝗶𝘁: - If a customer hits the same billing error twice, that’s not bad luck, it’s a broken loop. - If frontline staff keep hacks and workarounds to themselves, that’s a missed loop. - If leadership only hears what’s escalated, that’s a distorted loop. 𝗦𝗲𝗿𝘃𝗶𝗰𝗲 𝗱𝗲𝘀𝗶𝗴𝗻 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗶𝘀 𝗷𝘂𝘀𝘁 𝘁𝗵𝗲𝗮𝘁𝗲𝗿. 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝘄𝗶𝘁𝗵𝗼𝘂𝘁 𝗳𝗲𝗲𝗱𝗯𝗮𝗰𝗸 𝗮𝗿𝗲 𝗱𝗲𝘀𝘁𝗶𝗻𝗲𝗱 𝘁𝗼 𝗳𝗮𝗶𝗹. 𝗪𝗵𝗮𝘁 𝗰𝗮𝗻 𝘆𝗼𝘂 𝗱𝗼 𝘁𝗼𝗱𝗮𝘆? ✅ Embed feedback into your journeys—not after them ✅ Make insights operational, not optional ✅ Connect customer data to employee experience ✅ Design loops at every level—from micro-interactions to org-wide transformation 𝗬𝗼𝘂 𝗰𝗮𝗻’𝘁 𝗶𝗺𝗽𝗿𝗼𝘃𝗲 𝘄𝗵𝗮𝘁 𝘆𝗼𝘂 𝗱𝗼𝗻’𝘁 𝗹𝗶𝘀𝘁𝗲𝗻 𝘁𝗼. 𝗔𝗻𝗱 𝘆𝗼𝘂 𝗰𝗮𝗻’𝘁 𝗹𝗲𝗮𝗱 𝘄𝗵𝗮𝘁 𝘆𝗼𝘂 𝗰𝗮𝗻’𝘁 𝗹𝗲𝗮𝗿𝗻 𝗳𝗿𝗼𝗺. #ServiceDesign #OrganizationalDesign #BusinessDesign #SystemsDesign #Research