How to Implement Iterative Improvement in AI Strategies

Last week, I described four design patterns for AI agentic workflows that I believe will drive significant progress: Reflection, Tool use, Planning and Multi-agent collaboration. Instead of having an LLM generate its final output directly, an agentic workflow prompts the LLM multiple times, giving it opportunities to build step by step to higher-quality output. Here, I'd like to discuss Reflection. It's relatively quick to implement, and I've seen it lead to surprising performance gains. You may have had the experience of prompting ChatGPT/Claude/Gemini, receiving unsatisfactory output, delivering critical feedback to help the LLM improve its response, and then getting a better response. What if you automate the step of delivering critical feedback, so the model automatically criticizes its own output and improves its response? This is the crux of Reflection. Take the task of asking an LLM to write code. We can prompt it to generate the desired code directly to carry out some task X. Then, we can prompt it to reflect on its own output, perhaps as follows: Here’s code intended for task X: [previously generated code] Check the code carefully for correctness, style, and efficiency, and give constructive criticism for how to improve it. Sometimes this causes the LLM to spot problems and come up with constructive suggestions. Next, we can prompt the LLM with context including (i) the previously generated code and (ii) the constructive feedback, and ask it to use the feedback to rewrite the code. This can lead to a better response. Repeating the criticism/rewrite process might yield further improvements. This self-reflection process allows the LLM to spot gaps and improve its output on a variety of tasks including producing code, writing text, and answering questions. And we can go beyond self-reflection by giving the LLM tools that help evaluate its output; for example, running its code through a few unit tests to check whether it generates correct results on test cases or searching the web to double-check text output. Then it can reflect on any errors it found and come up with ideas for improvement. Further, we can implement Reflection using a multi-agent framework. I've found it convenient to create two agents, one prompted to generate good outputs and the other prompted to give constructive criticism of the first agent's output. The resulting discussion between the two agents leads to improved responses. Reflection is a relatively basic type of agentic workflow, but I've been delighted by how much it improved my applications’ results. If you’re interested in learning more about reflection, I recommend: - Self-Refine: Iterative Refinement with Self-Feedback, by Madaan et al. (2023) - Reflexion: Language Agents with Verbal Reinforcement Learning, by Shinn et al. (2023) - CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing, by Gou et al. (2024) [Original text: https://lnkd.in/g4bTuWtU ]

I’ve had the chance to work across several #EnterpriseAI initiatives esp. those with human computer interfaces. Common failures can be attributed broadly to bad design/experience, disjointed workflows, not getting to quality answers quickly, and slow response time. All exacerbated by high compute costs because of an under-engineered backend. Here are 10 principles that I’ve come to appreciate in designing #AI applications. What are your core principles? 1. DON’T UNDERESTIMATE THE VALUE OF GOOD #UX AND INTUITIVE WORKFLOWS Design AI to fit how people already work. Don’t make users learn new patterns — embed AI in current business processes and gradually evolve the patterns as the workforce matures. This also builds institutional trust and lowers resistance to adoption. 2. START WITH EMBEDDING AI FEATURES IN EXISTING SYSTEMS/TOOLS Integrate directly into existing operational systems (CRM, EMR, ERP, etc.) and applications. This minimizes friction, speeds up time-to-value, and reduces training overhead. Avoid standalone apps that add context-switching or friction. Using AI should feel seamless and habit-forming. For example, surface AI-suggested next steps directly in Salesforce or Epic. Where possible push AI results into existing collaboration tools like Teams. 3. CONVERGE TO ACCEPTABLE RESPONSES FAST Most users have gotten used to publicly available AI like #ChatGPT where they can get to an acceptable answer quickly. Enterprise users expect parity or better — anything slower feels broken. Obsess over model quality, fine-tune system prompts for the specific use case, function, and organization. 4. THINK ENTIRE WORK INSTEAD OF USE CASES Don’t solve just a task - solve the entire function. For example, instead of resume screening, redesign the full talent acquisition journey with AI. 5. ENRICH CONTEXT AND DATA Use external signals in addition to enterprise data to create better context for the response. For example: append LinkedIn information for a candidate when presenting insights to the recruiter. 6. CREATE SECURITY CONFIDENCE Design for enterprise-grade data governance and security from the start. This means avoiding rogue AI applications and collaborating with IT. For example, offer centrally governed access to #LLMs through approved enterprise tools instead of letting teams go rogue with public endpoints. 7. IGNORE COSTS AT YOUR OWN PERIL Design for compute costs esp. if app has to scale. Start small but defend for future-cost. 8. INCLUDE EVALS Define what “good” looks like and run evals continuously so you can compare against different models and course-correct quickly. 9. DEFINE AND TRACK SUCCESS METRICS RIGOROUSLY Set and measure quantifiable indicators: hours saved, people not hired, process cycles reduced, adoption levels. 10. MARKET INTERNALLY Keep promoting the success and adoption of the application internally. Sometimes driving enterprise adoption requires FOMO. #DigitalTransformation #GenerativeAI #AIatScale #AIUX

Discovering repeatable workflows that incorporate LLMs to build highly complex things safely and efficiently feels completely different from any tool, product, or process that I've ever adopted. I'd recommend starting small. Acquire licenses for your favorite LLM, establish policies around proprietary data (including data preparation), and motivate your team to deliver outcomes more quickly and accurately. Over time, build and optimize dependable workflows around the prompts you created! "Prompt journaling" is a super powerful technique. Once you've discovered and implemented dependable workflows using Generative AI - then consider going the integration route. Understand, which of your systems would benefit most from an integration with your favorite model; or acquire an industry specific LLM product to continue down your automation journey. #artificialintelligence #promptjournaling #automation

The legal industry needs a more thoughtful approach to generative AI. Our customers are reporting larger gaps between what a tech company says they can do with AI vs. what they can actually do. Take these 7 steps to get the most out of your AI pilot: 1. Prepare a roadmap for what your legal department would like to achieve with 1-year, 3-year, and 5-year benchmarks — since almost all legal tasks can benefit from language models, pick a first use-case that will have the biggest impact or ROI. 2. Define what success would mean for you with this use-case. Ideally in a way that you can tie to objective metric-oriented results or outcomes. 3. Break the problem into pieces. Try to figure out how useful the AI you’re testing will be for this use-case in its broken down pieces. Test several language models to get a gold standard metric of quality on each component. 4. Prepare an evaluation dataset, so you know “this is in the input” and “this is the desired output” 5. Discover what the models you’re testing can do with no training (’zero shot’). Establish a baseline. 6. See how the models improve with low investment techniques like prompt engineering or “few shot” training where you embed a few examples into your prompt. Try to get numbers on the board. Are you achieving the success metrics you were shooting for? 7. Explore how higher-investment strategies like fine-tuning would improve the output. Although it can be tempting to let FOMO and excitement around generative AI push us to move fast, the companies who take the time to implement a thoughtful approach are in a better position to benefit the most from AI. Don’t let ROI become an afterthought.

This is a great article to guide companies at the early stages of implementing Gen AI solutions. With Gen AI on the horizon, the spotlight isn't just on innovation—it's on our data. An overwhelming 80% of data leaders recognize its transformative potential, yet a stark disconnect lies in the readiness of our data environments. Only a minuscule 6% have operational Gen AI applications. The call to action is evident: for Gen AI to redefine our future, the foundation starts with high-quality, meticulously curated data. Organizations must create a data environment that supports and enhances the capabilities of Gen AI, turning it into a critical asset for driving innovation and business growth. Laying a solid data foundation for unlocking the full potential of Gen AI involves a well-thought-out approach: 1—Assess Data Quality: Begin by thoroughly assessing current data quality. Identify gaps in accuracy, completeness, and timeliness. 2 - Data Integration and Management: Integrate disparate data sources to create a unified view. Employ robust data management practices to ensure data consistency and accessibility. 3 - Curate and Annotate Data: Ensure relevance and annotate it to enhance usability for Gen AI models. 4 - Implement Data Governance: Establish a robust data governance framework to maintain data integrity, security, and compliance to foster data sharing and collaboration. 5 - Invest in Scalable Infrastructure: Build or upgrade to a data infrastructure that can scale future Gen AI applications. This includes cloud storage, powerful computing resources, and advanced data processing capabilities. 6—Upskill Your Team: Ensure the technical team has the necessary skills to manage, analyze, and leverage data to build Gen AI solutions. 7 Pilot and Scale: To test and refine your approach, start with pilot projects. Use these learnings to scale successful initiatives across the organization. 8 - Continuous Improvement: Gen AI and data landscapes are evolving rapidly. Establish processes for ongoing data evaluation and model training to adapt to new developments and insights.

People are stubborn. AI is proving not to be. Imagine a world where we create AI Workspaces of egoless agents who, beyond anything else, self-improve and adjust. These AI Workspaces are not just about automating tasks; it's about creating an ecosystem where AI agents continuously communicate, iterate, and evolve – essentially, a living organism that learns and improves autonomously. A kind of utopic company that had only ever been a matter of imagination for C-suite executives. Here's an example of how it would work: 🎨 Design Phase: An AI designer analyzes trends and preferences to create varied design prototypes for different user segments. 🛠️ Engineering Handoff: AI engineers, understanding design intent and user needs, optimize code efficiently, ensuring seamless transition. 🔍 Quality Assurance: AI algorithms in QA dynamically test products, adjusting parameters based on real-time data for comprehensive coverage. 📈 Product Management: An AI product manager aligns product strategies with market trends, competitor analysis, and user feedback. 📣 Marketing and Sales: AI-driven marketing and sales analyze data, predict market response, and dynamically adjust strategies based on feedback. Feedback Loop and Iteration: Market performance feedback flows back to the AI designer, informing improvements with rich, contextual insights. With each cycle, AI systems refine, learning from past outcomes to enhance product design and market strategies. 🔁 This AI-driven feedback loop shatters traditional, linear workflows, creating a hyper-adaptive system that redefines product creation. It blurs the boundaries between development, analysis, and consumer feedback, crafting products that resonate deeply with market demands at unprecedented speed. In the AI Workspace, humans evolve from workers to orchestrators, sculpting AI workflows and directing the narrative. We set the stage for AI's performance, dictating the ordering and steps of its workflow, shaping outcomes with strategic foresight. As curators of this intelligent ecosystem, our role pivots to innovation and high-level strategy, while AI executes with a precision and adaptability we can't match. Interestingly, this shift might be the death of entry-level and middle management roles. We might be headed towards black and white employment. You’re either dictating AI workflows, or cashing in unemployment cheques. AI Workspaces herald the dawn of a new corporate species: ultra-agile, antifragile, and perpetually evolving entities. They represent a future where technology is not just a tool but an entire species governed by the same principles of natural selection. 🧬 What do you think? #AIWorkspaces #FeedbackLoopI #FutureOfWork #BusinessAutomation #AIinBusiness #Innovation #techevolution #StrategicAI #AdaptiveTechnology #career #productdevelopment