How to Measure ROI in Automation

Kat Shoa - great question - how do you measure the horizontal ROI (ex of AI in email, call transcription, etc)? This is such a smart distinction - and you're right that horizontal ROI is trickier to measure precisely because it's so distributed. Here's how I think about it but Brice Challamel, Greg Shove, Shruthi Shetty, Tony Gentilcore, Section or Tony Hoang may have more to add ... 𝗧𝗵𝗲 𝗛𝗼𝗿𝗶𝘇𝗼𝗻𝘁𝗮𝗹 𝗥𝗢𝗜 𝗖𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 When AI touches everyone's email, transcription, or document creation, the impact gets diffused across every workflow. You can't easily isolate "the AI effect" because it becomes infrastructure - like trying to measure the ROI of electricity or internet connectivity. 𝗧𝗵𝗲 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗧𝗿𝗶𝗰𝗸 𝗛𝗲𝗿𝗲'𝘀 𝘄𝗵𝗮𝘁 𝘄𝗼𝗿𝗸𝘀: Create control groups. Roll out horizontal AI to Department A but not B for 90 days. Measure productivity, employee satisfaction, and output quality differences. The delta is your horizontal ROI. 𝗧𝗵𝗿𝗲𝗲-𝗟𝗮𝘆𝗲𝗿 𝗠𝗲𝗮𝘀𝘂𝗿𝗲𝗺𝗲𝗻𝘁 𝗔𝗽𝗽𝗿𝗼𝗮𝗰𝗵 𝗟𝗮𝘆𝗲𝗿 𝟭: 𝗔𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲 𝗧𝗶𝗺𝗲 𝗘𝗰𝗼𝗻𝗼𝗺𝗶𝗰𝘀 Start with the math everyone can understand: If 500 employees save 30 minutes daily on email/transcription, that's 250 hours per day. At an average wage of $50/hour, that's $12,500 daily or $3.25M annually. Simple, defensible baseline. 𝗟𝗮𝘆𝗲𝗿 𝟮: 𝗤𝘂𝗮𝗹𝗶𝘁𝘆 𝗠𝘂𝗹𝘁𝗶𝗽𝗹𝗶𝗲𝗿𝘀 But the real value isn't just time saved - it's what people do with freed cognitive capacity. Track: - Meeting quality scores (when transcription handles notes, do people participate more?) - Email response rates and customer satisfaction - Cross-functional collaboration frequency 𝗟𝗮𝘆𝗲𝗿 𝟯: 𝗖𝗼𝗺𝗽𝗼𝘂𝗻𝗱 𝗡𝗲𝘁𝘄𝗼𝗿𝗸 𝗘𝗳𝗳𝗲𝗰𝘁𝘀 This is where horizontal AI gets interesting. When everyone has better email/transcription, the entire communication system improves. Measure: - Decision speed (time from question to action) - Information cascade velocity (how fast insights spread) - Coordination overhead reduction 𝗕𝗼𝘁𝘁𝗼𝗺 𝗟𝗶𝗻𝗲 Horizontal ROI is often your biggest ROI story - but you have to measure it at the system level, not the individual level. Think platform economics, not feature economics. Would love other thoughts on above. And if needed Lanai team is happy to deep w/ folks who are working to get more data-driven on delivering measurable impact with AI tooling.

Vendors say, “AI coding tools are writing 50% of Google’s code.” I say, “Autocomplete or IntelliSense was writing about 25% of Google’s code, and AI made it twice as effective.” When it comes to measuring AI’s ROI, real-world benchmarks are critical. Always compare the current state to the future state to calculate value instead of just looking at the future state. Most companies are overjoyed to see that AI coding tools write 30% of their code, but when they realize that vanilla IDEs with basic autocomplete could do 25%, the ROI looks less impressive. 5% rarely justifies the increased licensing and token costs. That’s the reality I have found with about half of the AI tools I pilot with clients. They work, but the improvement over the current state isn’t worth their price. I have used the same method to measure ROI for almost a decade. 1️⃣ Benchmark the current process performance using value outcomes. 2️⃣ Propose a change to the current process that introduces technology/new technology into the workflow. 3️⃣ Quantify the expected change in outcomes and value delivered with the new process/workflow. 4️⃣ Make the update and measure actual outcomes. If there’s a difference between expected vs. actual, find the root cause and fix it if possible. Measuring AI ROI is simple with the right framework. It’s also easier to help business leaders make better decisions about technology purchases, customer-facing features, and internal productivity initiative selection. I would rather see a benchmark like, percentage of code generated from text prompts vs. the percentage of code recommended by autocomplete. That benchmarks the reengineered process against the old one. AI process reengineering (AI tools augmenting people performing an optimized workflow) is where I see the greatest ROI. Shoehorning AI tools into the current process typically delivers a fraction of the potential ROI.

How to Measure AI ROI: A Step-by-Step Guide That Actually Works Most companies waste millions on AI without knowing if it works. Looking to maximize your AI investments? Here's your roadmap to success: Step 1: Define Clear Success Metrics • Revenue impact • Cost savings • Time saved • Customer satisfaction scores • Employee productivity gains Step 2: Implement the AI Decision Scorecard • Compliance checks • Quality assessment • Employee experience • Business impact measurement Step 3: Set Baseline Measurements • Current performance metrics • Cost of operations • Time per task • Error rates • Customer feedback Step 4: Track Progress • Weekly data collection • Monthly progress reviews • Quarterly ROI calculations • Stakeholder feedback • Performance adjustments Step 5: Scale What Works • Document successful use cases • Share wins across teams • Replicate winning patterns • Train more users • Expand implementation The Truth: Only 22% of companies measure AI ROI effectively. Don't be part of that statistic. Remember: If you can't measure it, you can't improve it. Ready to transform your AI investments into real results? Share your biggest AI measurement challenge below 👇

Here's a Gemini deep research prompt that helps quantify ROI for AI—especially if you're building features that save users time. We're using this at HubSpot to estimate "time saved" across our AI product suite (Agents, Copilot, and 100s of embedded features). It's already helped us calculate how much effort is being offset by tools like Breeze Content Agent or Customer Agent. This prompt will: - Analyze any AI feature - Identify its job-to-be-done - Estimate the manual time that job would take - Estimate how much of that time the AI saves - Justify the estimates with clear reasoning Our customers don't want AI that's novel—but necessary. This is a powerful way to show what they're getting when they choose Breeze. Here's the prompt: "You are a deep research model tasked with helping a product manager at [Insert company name] quantify ROI for AI features." Context: [Add context on the AI products you offer] We are building an out-of-the-box analytics product that helps customers understand their AI usage and ROI. The core ROI metric is "time saved." We define time saved as: Estimated time (in hours) that would have been spent doing the task manually × % of the task completed by the AI. Approach: We've already modeled this metric for a few AI features by combining SME interviews and LLM-based research. See examples below: ✅ Prior Examples: [Insert examples of feature job to be done manual hours % time offset] ❓Your Task:  Given a catalog of additional AI features (attached separately), please: For each feature in the catalog: 1. Identify the likely job-to-be-done (JTBD). 2. Estimate manual hours required to perform the job. 3. Estimate the % of time offset by the AI (i.e., how much of the manual effort the AI completes accurately). 4. Justify your estimates with reasoning (cite analogies or research if possible). Output format: pgsql Copy Edit Feature Name: [Insert feature name] Job To Be Done: [Insert JTBD] Estimated Manual Hours: [X hrs] % Time Offset by AI: [X%] Rationale: [2–4 sentences summarizing assumptions, proxies, or analogies used] If a feature is ambiguous or lacks clarity, make a reasonable assumption about its intended use case and state that assumption clearly in your rationale. Each Feature Name should be analyzed individually.

𝑵𝒆𝒘 𝒑𝒐𝒔𝒕 𝒔𝒆𝒓𝒊𝒆𝒔 -- 𝑮𝒆𝒏 𝑨𝑰 𝒇𝒐𝒓 𝑵𝒆𝒕𝒘𝒐𝒓𝒌𝒔. 𝑷𝒐𝒔𝒕 6/7 Setting Clear Objectives for AI Integration = Measuring ROI When implementing AI initiatives, it's crucial to ➡ establish clear, measurable objectives, ➡seamlessly integrate AI into existing processes, ➡ continuously measure ROI to ensure alignment with business goals. 𝐂𝐥𝐞𝐚𝐫 𝐃𝐞𝐟𝐢𝐧𝐢𝐭𝐢𝐨𝐧 𝐨𝐟 𝐎𝐛𝐣𝐞𝐜𝐭𝐢𝐯𝐞𝐬 To ensure that AI initiatives are successful, start by setting clear, measurable objectives that align with your overall business goals: ✅ Setting targets for cost reduction through automation and optimization. For instance, a McKinsey report indicates that AI-driven predictive maintenance can reduce maintenance costs by up to 20% and cut unplanned downtime by 50%. ✅Enhancing customer experience by leveraging AI for personalized recommendations, chatbots, and 24/7 support. Gartner predicts that by 2025, 80% of customer service interactions will be handled by AI, leading to faster response times and higher customer satisfaction. ✅Generating new revenue streams by using AI to identify market opportunities and develop innovative products. PwC studies show that AI could contribute up to $15.7 trillion to the global economy by 2030, highlighting its potential for creating new business opportunities. 𝐏𝐫𝐨𝐜𝐞𝐬𝐬 𝐈𝐧𝐭𝐞𝐠𝐫𝐚𝐭𝐢𝐨𝐧 For AI to deliver its maximum value, it needs to be seamlessly integrated into existing business processes: ✅Mapping out existing workflows to identify areas where AI can be most beneficial, such as repetitive, time-consuming tasks that can be automated. ✅Designing a strategic integration plan that minimizes disruptions while maximizing the benefits of AI technologies. Start with pilot projects to test the integration process and refine your approach based on feedback and initial results. 𝐌𝐞𝐚𝐬𝐮𝐫𝐞𝐦𝐞𝐧𝐭 𝐨𝐟 𝐑𝐎𝐈 To justify AI investments, it's essential to establish and continuously monitor metrics that measure the return on investment: ✅Tracking direct financial gains, such as cost savings from automation, increased sales from personalized marketing, or new revenue streams from AI-driven products. ✅Measuring indirect benefits like improvements in customer satisfaction, operational efficiency, and employee productivity. For example, AI can streamline customer service operations, leading to faster response times and higher customer satisfaction ratings. ✅Implementing a robust monitoring system to continuously track these metrics, regularly evaluating the success of AI implementations, and making necessary adjustments to optimize performance and outcomes. This structured methodology helps organizations harness the full potential of AI, driving both innovation and efficiency. What would you add?

“𝐑𝐚𝐬𝐡𝐢𝐦, if I can’t show ROI, how do I justify investment in AI?” AI ROI isn’t about long-range fantasy. It’s about operational wins today that compound over time. 🔹 1. 𝐐𝐮𝐚𝐧𝐭𝐢𝐟𝐲 𝐭𝐡𝐞 𝐞𝐟𝐟𝐢𝐜𝐢𝐞𝐧𝐜𝐲 𝐝𝐢𝐯𝐢𝐝𝐞𝐧𝐝. Measure cost-per-task reduction, team velocity improvements, and SLA acceleration. Time is money-track both. 🔹 2. 𝐂𝐨𝐧𝐧𝐞𝐜𝐭 𝐀𝐈 𝐭𝐨 𝐫𝐞𝐯𝐞𝐧𝐮𝐞 𝐝𝐫𝐢𝐯𝐞𝐫𝐬. Are sellers reaching customers faster? Is marketing personalizing faster? Is CS pre-empting churn? Align usage to business KPIs. 🔹 3. 𝐁𝐮𝐢𝐥𝐝 𝐚 𝐛𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤 𝐭𝐡𝐚𝐭 𝐞𝐯𝐨𝐥𝐯𝐞𝐬. Start with time savings. Expand to cost savings. Mature into revenue uplift. Create an ROI path that scales. 💡 AI ROI isn’t a one-time report. It’s a continuous improvement curve. 👇 What ROI signals are you tracking from your AI efforts? #BytesfromRashim #AI #AIADOPTION