Key Differences Between Traditional and AI Product Management Roles

Does developing product vision & strategy really differ for AI products, compared to traditional software product management? 📣 YES 📣 ! Some keys to navigating the differences and pitfalls to avoid, with examples from my time at Alexa AI: 1️⃣ Map AI to 10x Value: Look for opportunities where AI capabilites can significantly reduce friction or provide impossible-with-traditional-methods insights. 2️⃣ Set Realistic Launch Criteria: Perfect accuracy isn't always the goal. Use existing systems (even human ones) as benchmarks. 3️⃣ Balance Tech & Strategy: Always tie AI initiatives to user/business KPIs. Ask, "What would have to be true for this to be a good idea?" 4️⃣ Measure Holistically: Consider a constellation of metrics beyond primary KPIs - include model performance, ethical considerations, and operational metrics. 5️⃣ Prioritize Ethics: Conduct comprehensive stakeholder analysis and maintain a dynamic risk register throughout the project lifecycle. 6️⃣ Communicate Clearly: Make risks tangible to stakeholders. Include potential negative PR headlines in requirements docs and show real worst-case scenario data. 💡 Key Takeaway: In AI PM, your strategy must be as adaptive and learning-oriented as the systems you're building. Embrace uncertainty, plan for ethical implications, and most of all - keep the end-user at the center of your vision. We are PMs first and foremost, after all - we just happen to build with AI :D

Product managers & designers working with AI face a unique challenge: designing a delightful product experience that cannot fully be predicted. Traditionally, product development followed a linear path. A PM defines the problem, a designer draws the solution, and the software teams code the product. The outcome was largely predictable, and the user experience was consistent. However, with AI, the rules have changed. Non-deterministic ML models introduce uncertainty & chaotic behavior. The same question asked four times produces different outputs. Asking the same question in different ways - even just an extra space in the question - elicits different results. How does one design a product experience in the fog of AI? The answer lies in embracing the unpredictable nature of AI and adapting your design approach. Here are a few strategies to consider: 1. Fast feedback loops : Great machine learning products elicit user feedback passively. Just click on the first result of a Google search and come back to the second one. That’s a great signal for Google to know that the first result is not optimal - without tying a word. 2. Evaluation : before products launch, it’s critical to run the machine learning systems through a battery of tests to understand in the most likely use cases, how the LLM will respond. 3. Over-measurement : It’s unclear what will matter in product experiences today, so measuring as much as possible in the user experience, whether it’s session times, conversation topic analysis, sentiment scores, or other numbers. 4. Couple with deterministic systems : Some startups are using large language models to suggest ideas that are evaluated with deterministic or classic machine learning systems. This design pattern can quash some of the chaotic and non-deterministic nature of LLMs. 5. Smaller models : smaller models that are tuned or optimized for use cases will produce narrower output, controlling the experience. The goal is not to eliminate unpredictability altogether but to design a product that can adapt and learn alongside its users. Just as much as the technology has changed products, our design processes must evolve as well.

AI is challenging some of the fundamentals of product management. Remember the good, old 2 x 2 prioritization framework? With AI there are new dimensions that must be added to it. Impact and Cost are what they always were; and even the way to measure them does not have to change. However, in the age of AI, product managers need to think about three more things: 1️⃣ Time to Launch: Time has always been super important in the world of product. The sooner you start getting feedback, the better off you would be in your attempt to find the product market fit. With AI, however, time takes a whole different importance. First, the sooner you start generating real world data, the stronger your models will become. Second, and perhaps more importantly, the sooner you can confirm the fundamental assumptions about your modeling, the less likely would you be to go in the wrong direction. For traditional software, time was more or less directly proportional with cost, but not so with AI, e.g., compare GPU-intensive but well understood use case like image classification with, say, a bespoke system to tag leads. 2️⃣ Solution Inaccuracy: Unlike deterministic programing, AI comes with the fact that it may not always work. So would you prioritize a quick solution that is right half the time, or a complex solution that is right 90% of the time. The answer is not trivial, ever, but this dimension now is a strong consideration in terms of picking and choosing projects. The kicker? For AI systems you may not even know its accuracy in advance. You can guess, at best. This is why many software teams still prefer predictable lower impact legacy solutions over AI. This is also why quick, agile implementations with lower time to launch become instrumental. 3️⃣ Cost of Failure: The final dimension is to think about what happens if AI gets it wrong, even for highly accurate systems. In a B2C app with a quick feedback loop it doesn’t really matter, because the user will immediately provide a corrective signal and things will be fine. For example - the text auto complete feature in Gmail. Otoh, this can get very tricky for enterprise applications in regulated contexts. A wrong decision can expose the company to law suits or significant financial losses. For example - banking and/or trading systems. So, instead of the 2x2 it’s time to consider the Spider Chart while prioritizing. Plot everything and work from the center outwards. It will be very interesting to see how PMs navigate the inherent trade-offs. Are there dimensions that you will add to the spider? #ai #generatieveai #productmanagement