Multi-Agent Prompting Strategies

🚀 My favorite prompting trick that you probably haven't seen: Simulating Agentic Behavior With a Prompt 🤖 After spending now likely thousands of hours prompting #LLMs, one thing I've found that can vastly improve the quality of outputs is something I haven't seen talked about much. ✨ "Instantiate two agents competing to find the real answer to the given problem and poke holes in the other agent's answers until they agree, which they are loathe to do." ✨ This works especially well with #CLAUDE3 and #Opus. For a more advanced version that often works even better: ✨"Instantiate two agents competing to find the real answer and poke holes in the other's answer until they agree, which they are loathe to do. Each agent has unique skills and perspective and thinks about the problem from different vantage points. Agent 1: Top-down agent Agent 2: Bottom-up agent Both agents: Excellent at the ability to think counter factually, think step by step, think from first principles, think laterally, think about second order implications, are highly skilled at simulating in their mental model and thinking critically before answering, having looked at the problem from many directions." ✨ This often solves the following issues you will encounter with LLMs: 1️⃣ Models often will pick the most likely answer without giving it proper thought, and will not go back to reconsider. With these kinds of prompts, the second agent forces this, and the result is a better-considered answer. 2️⃣ Continuing down the wrong path. There's an inertia to an answer, and the models can often get stuck, biased toward a particular kind of wrong answer or previous mistake. This agentic prompting improves this issue significantly. 3️⃣ Overall creativity of output and solution suggestions. Having multiple agents considering solutions results in the model considering solutions that might otherwise be difficult to elicit from the model. If you haven't tried something like this and have a particularly tough problem, try it out and let me know if it helps!

Leveraging Agent-Based Approaches for Complex Tasks There's growing buzz around how agent-based approaches can help AI tackle complex tasks that seem beyond the reach of today's LLMs. Here are two promising methods (all references in the comments): Role-Playing Agents: Traditional machine translators often fall short in complex translation tasks, such as translating books—something even highlighted in a recent NYT article. A recent paper introduces a multi-agent approach to mimic organizational task execution. They created specialized AI agents acting as a CEO, senior editors, junior editors, translators, localization specialists, and proofreaders. The senior editor agent sets editorial standards, junior editor agents plan the workflow, localization specialists handle cultural references, and proofreader agents focus on proofreading. This collaborative effort, with editor agents critiquing AI translations against editorial guidelines, resulted in AI translations preferred over human ones by evaluators. I've also found critique agents invaluable in my work. Increasing Agent Count: For challenging tasks, employing multiple agents to perform the same task and then combining their work (e.g., through averaging or voting) can enhance performance. A recent study applied an ensemble of LLM models to reasoning and code generation tasks, finding that performance improved with more models. Remarkably, an ensemble of 15 Llama2-70B models outperformed a single GPT-3.5 model. In my own work on evaluating natural language generation, I found that an ensemble of 10 LLM evaluators surpassed a single evaluator. Note: While these approaches significantly boost performance, they also increase costs and slow down the system.

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]