Evaluating Change Management Implementation Success

Most change initiatives are measured by one number: Adoption. Did people start using the new system? Did they attend the training? Did they log in? But just because something was adopted doesn’t mean the change worked. Adoption tells you if people used it. It doesn’t tell you how well they’re using it or whether it made anything better. To really measure change success, you need to go deeper: – Is behavior actually different? Are people making decisions in a new way? Are old habits starting to fade? – Is performance improving? Has the change helped teams deliver better results, faster service, fewer errors, or stronger collaboration? – Is the change sustainable? Are people still using the new way of working 3, 6, 12 months later or did things quietly go back to how they were? – Do people understand why the change matters? Real change sticks when people connect it to their purpose, not just their process. Success isn’t just about launch day. It’s about what happens after, when the excitement fades and the real work begins.

Last month, our AI tool adoption rate reached 62.5% among 40 engineers. But that number only tells part of the story. When I shared our change management approach and experimentation framework in previous posts, many of you asked: "How do you actually measure success?" The answer? We have built a comprehensive tracking system that focuses on encouragement, rather than enforcement. 1. Make it visible everywhere. We keep AI adoption front-of-mind through: Bi-weekly NPS surveys (54.5 current score) Monthly Community of Practice meetings Active Slack channel for sharing wins and learnings Real-time usage dashboards are shared team-wide The key insight: visibility drives curiosity, which in turn drives adoption. 2. Track both tools AND outcomes. We monitor two distinct categories: - Agentic Development tools (Copilot, Claude, Cursor) - Conversational AI (ChatGPT, Gemini, Claude) But here's what most teams miss—we also track work outcomes by tagging Jira tickets as "agentic_success" or "agentic_failure." This connects tool usage to actual impact. 3. Focus on insights, not enforcement. Our bi-weekly surveys don't just ask "did you use AI?" They capture: - Which specific tools do teams prefer - Key insights from their experiments - Barriers preventing adoption - Success stories worth sharing The result? 4.8M+ tokens used, 678% growth month-over-month, and most importantly—engineers actively sharing what works. Remember: this isn't about forcing adoption through metrics. It's about creating transparency that encourages experimentation. The dashboard becomes a conversation starter, not a performance review. What metrics have you found most valuable for tracking innovation adoption in your teams? P.S. Links to the change management and experimentation posts in the comments for those catching up on the series. #AIAdoption #EngineeringLeadership #TechTransformation #AgileMetrics

How will we know we've been successful? Too often, in organizational transformation efforts, the answer to that question is in the eye of the beholder. Rather than arguing about who's perspective is "right", however, it can be useful to think about organizational change success as multi-dimensional. Change is complex, after all. Our measures of it should reflect that reality, without being overly complicated. For me, the three buckets of achievement, completion, and acceptability fit that bill. These come directly from the work of Susan Miller on implementing strategic decisions - please see the comments for a link. ➡ Achievement: Gauge how close you've come to creating meaningful results from the change you are implementing. This helps to ensure you don't mistake activity for outcomes. That said, Rome wasn't built in a day, so it's wise to look for signs of learning first before evaluating performance. ➡ Acceptability: Get feedback on stakeholder satisfaction regarding the change solution and process. People's current experience with change can shape their view of future changes. So, getting better here can bring both short and long-term dividends. ➡ Completion: Assess how well you execute. Bringing about change in organizations can be an intensive and expensive undertaking. Managing schedule, scope, and budget effectively is a critical. Additionally, some research indicates that people's perspectives of a given change are influenced by their belief in the organization's ability to manage it competently. Using defined success dimensions such as these can build shared understanding of different perspectives and enable more robust conversations about the definition of success for any given change. Importantly, developing relevant measures and tracking progress on multiple dimensions can also support more nuanced learning about what aspects of change you are doing well and the best areas to target if you want to improve your organization's approach to change. If change is truly the only constant in organizations today, surely that's something worth knowing. #changemanagement #transformation #transformationalchange #organizationalchange #changeleadership