The Changing Nature of Performance Metrics With AI

Over the last year, I’ve seen many people fall into the same trap: They launch an AI-powered agent (chatbot, assistant, support tool, etc.)… But only track surface-level KPIs — like response time or number of users. That’s not enough. To create AI systems that actually deliver value, we need 𝗵𝗼𝗹𝗶𝘀𝘁𝗶𝗰, 𝗵𝘂𝗺𝗮𝗻-𝗰𝗲𝗻𝘁𝗿𝗶𝗰 𝗺𝗲𝘁𝗿𝗶𝗰𝘀 that reflect: • User trust • Task success • Business impact • Experience quality    This infographic highlights 15 𝘦𝘴𝘴𝘦𝘯𝘵𝘪𝘢𝘭 dimensions to consider: ↳ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆 — Are your AI answers actually useful and correct? ↳ 𝗧𝗮𝘀𝗸 𝗖𝗼𝗺𝗽𝗹𝗲𝘁𝗶𝗼𝗻 𝗥𝗮𝘁𝗲 — Can the agent complete full workflows, not just answer trivia? ↳ 𝗟𝗮𝘁𝗲𝗻𝗰𝘆 — Response speed still matters, especially in production. ↳ 𝗨𝘀𝗲𝗿 𝗘𝗻𝗴𝗮𝗴𝗲𝗺𝗲𝗻𝘁 — How often are users returning or interacting meaningfully? ↳ 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗥𝗮𝘁𝗲 — Did the user achieve their goal? This is your north star. ↳ 𝗘𝗿𝗿𝗼𝗿 𝗥𝗮𝘁𝗲 — Irrelevant or wrong responses? That’s friction. ↳ 𝗦𝗲𝘀𝘀𝗶𝗼𝗻 𝗗𝘂𝗿𝗮𝘁𝗶𝗼𝗻 — Longer isn’t always better — it depends on the goal. ↳ 𝗨𝘀𝗲𝗿 𝗥𝗲𝘁𝗲𝗻𝘁𝗶𝗼𝗻 — Are users coming back 𝘢𝘧𝘵𝘦𝘳 the first experience? ↳ 𝗖𝗼𝘀𝘁 𝗽𝗲𝗿 𝗜𝗻𝘁𝗲𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — Especially critical at scale. Budget-wise agents win. ↳ 𝗖𝗼𝗻𝘃𝗲𝗿𝘀𝗮𝘁𝗶𝗼𝗻 𝗗𝗲𝗽𝘁𝗵 — Can the agent handle follow-ups and multi-turn dialogue? ↳ 𝗨𝘀𝗲𝗿 𝗦𝗮𝘁𝗶𝘀𝗳𝗮𝗰𝘁𝗶𝗼𝗻 𝗦𝗰𝗼𝗿𝗲 — Feedback from actual users is gold. ↳ 𝗖𝗼𝗻𝘁𝗲𝘅𝘁𝘂𝗮𝗹 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱𝗶𝗻𝗴 — Can your AI 𝘳𝘦𝘮𝘦𝘮𝘣𝘦𝘳 𝘢𝘯𝘥 𝘳𝘦𝘧𝘦𝘳 to earlier inputs? ↳ 𝗦𝗰𝗮𝗹𝗮𝗯𝗶𝗹𝗶𝘁𝘆 — Can it handle volume 𝘸𝘪𝘵𝘩𝘰𝘶𝘵 degrading performance? ↳ 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗥𝗲𝘁𝗿𝗶𝗲𝘃𝗮𝗹 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 — This is key for RAG-based agents. ↳ 𝗔𝗱𝗮𝗽𝘁𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗦𝗰𝗼𝗿𝗲 — Is your AI learning and improving over time? If you're building or managing AI agents — bookmark this. Whether it's a support bot, GenAI assistant, or a multi-agent system — these are the metrics that will shape real-world success. 𝗗𝗶𝗱 𝗜 𝗺𝗶𝘀𝘀 𝗮𝗻𝘆 𝗰𝗿𝗶𝘁𝗶𝗰𝗮𝗹 𝗼𝗻𝗲𝘀 𝘆𝗼𝘂 𝘂𝘀𝗲 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗽𝗿𝗼𝗷𝗲𝗰𝘁𝘀? Let’s make this list even stronger — drop your thoughts 👇

Discussed how to measure "productivity" with Nick Wienholt yesterday. Here are some notes and observations: 1. We should plot the number of lines of code written by AI versus a human developer and drive towards 100% <<---- (NOT - DON'T do this) 2. We should see an increase in velocity when using AI and drive it as high as possible <<---- (NOT - DON'T do this) 3. We should see a decrease in the number of developers in terms of number and cost. <<---- (NOT - DON'T do this) We are seeing industry "leaders" (I'm looking at you facebook) talk about AI impact using all three. We have to offer better alternatives. We have to be explicit about what type of improvements we expect. We have to be explicit about what we RISK when doing that. If you know me at all, you know I think performance is a six-dimensional tug-of-war (see image). When we pull on any performance rope, we risk negatively impacting others. So, when we "change" a process (and AI coding is a change), we need to be aware of the impact. If there are gaps in our performance measurement, then we are likely blindly causing demise. Six dimensions sound like a lot. But, often one measure is an indicator for more than one dimension. I'm thinking about this NOW, so I don't have full answers yet, but here is my first draft: 1. Do the right stuff: When using AI, are we cherry picking AI-able work versus business outcome work? For me, are we doing work in the DESIRED order? (I call this the wrong-order-o-meter) 2, 3. Do Lots and Do Predictably: These go together. They are a cadence. The actual values indicate whether we are moving too fast or too slow. If we are doing the right stuff, then this just needs to be stable. For me, this is "Product releases per x"—just confirming that stuff is going out the door, not just being written. 4. Do it fast: This is where I hope we can have UI make an impact. It has to make it faster concerning customer impact (not just coding time impact). Ideas, features, and bug fixes spend a lot more time scheduled than being done. "Lead time reduction of delivered high priority features and fixes" (time from created to delivered, NOT started dev to finished dev). 5. Do it right: MAJOR guardrail. This has to measure whether what we delivered solved the real problem and didn't require remediation. I'm going to say we measure "Release rollbacks"—start with production, get that to zero, and then measure in a pre-prod environment as well. Mistakes happen, so zero is just as bad as too many. A stable rate not increasing (or decreasing) is my preferred approach here. 6. Sustainability (keep doing it): Traditionally, it was a people metric. Do the teams work and deliver at a sustainable pace? There is a system component to this, where technical debt is increasing fast. I think unintended breaking changes by AI are a key measure here. Perhaps Release Rollbacks are an indicator that even the strongest AI is unable to make safe changes to the system. Thoughts?

Managing Performance in the Age of AI The way we measure performance has always been tied to the nature of work — and that nature has changed dramatically over time. First era → Operational work Tasks were repetitive, well-defined, and measurable. Quantity mattered more than anything else (with a baseline of quality). It was simple: assign the task, check if it’s done, measure output. Performance was visible almost instantly. Second era → Knowledge work Things got more complex — expertise and judgment started to matter. Here, quality became far more important than quantity (though both still played a role). Businesses and clients were willing to pay a premium for quality. Timelines stretched, but that was fine if the final project met the standard. Performance measurement was still manageable: look at the outcome, assess quality, and you knew whether the person delivered. Third era → Novel work (today, in the age of AI) AI has changed the game — quality and quantity can now be generated at scale, on demand, 24×7. The differentiator isn’t how much you produce, or even how polished it looks — machines already excel at that. The differentiator is creativity: solving new problems in new ways. And that’s far trickier to measure. Creativity doesn’t run on predictable timelines. You can’t guarantee novelty just by working harder or longer. A person could put in endless hours and still not create something truly new. Which means performance measurement has become slower and riskier. Businesses may have to wait much longer before they know whether someone can deliver in this “novel work” world. So what’s the answer? I believe we need to shift focus away from lagging indicators (outputs, outcomes) and start looking at leading indicators — early signs that a person has the mindset and skills to thrive in this new environment. Here are some early indicators: 1. Work ethics – discipline, consistency, ownership of outcomes. 2. Contribution in meetings – do they bring valuable perspectives, even if you need to draw it out of them? 3. Comfort with conflicting thoughts – can they handle ambiguity and contradictions without stalling? 4. Contextual understanding – do they grasp what’s happening around them and shape solutions that actually fit the situation? 5. Articulation – can they communicate ideas with clarity? In tomorrow’s world, articulation will matter more than knowledge, because knowledge is already democratized. In short → managing performance in the AI era means measuring what machines cannot. Not just IQ, but also EQ and SQ. Not just end results, but the early signals of creativity, adaptability, and alignment. This is harder — no doubt. But it’s also necessary. Because without evolving our performance measurement, we’ll keep applying old metrics to a new world — and miss what really drives value today. What do you think — are leaders and organizations ready to rewire performance management for this reality?