How AI Affects Employment in Customer Support

🧠 Is Generative AI Just Cool, or Does It Really Have an Impact? That's the big debate in tech circles these days. A study led by researchers from Stanford University, MIT, and the National Bureau of Economic Research (NBER) sheds light on this question by examining the real-world impact of deploying generative AI in a customer support environment. Their analysis offers empirical evidence on how AI tools, specifically those based on OpenAI's GPT models, are transforming customer service operations at a Fortune 500 software company. The researchers employed a mix of methodologies: a randomized control trial (RCT) and a staggered rollout, encompassing around 5,000 agents over several months. By analyzing 3 million customer-agent interactions, the study assessed metrics such as resolutions per hour, handle time, resolution rates, and customer satisfaction (Net Promoter Score). To understand the AI's impact over time, dynamic difference-in-differences regression models were used. Here is what they found: 1. 𝐒𝐢𝐠𝐧𝐢𝐟𝐢𝐜𝐚𝐧𝐭 𝐁𝐨𝐨𝐬𝐭 𝐢𝐧 𝐏𝐫𝐨𝐝𝐮𝐜𝐭𝐢𝐯𝐢𝐭𝐲: The AI tool led to a 13.8% increase in the number of customer queries resolved per hour, particularly benefiting less experienced agents. 2. 𝐍𝐚𝐫𝐫𝐨𝐰𝐢𝐧𝐠 𝐭𝐡𝐞 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞 𝐆𝐚𝐩: AI tools accelerated the learning curve for newer agents, allowing them to reach the performance levels of seasoned employees more quickly. 3. 𝐈𝐦𝐩𝐫𝐨𝐯𝐞𝐝 𝐂𝐮𝐬𝐭𝐨𝐦𝐞𝐫 𝐒𝐚𝐭𝐢𝐬𝐟𝐚𝐜𝐭𝐢𝐨𝐧: The AI deployment resulted in higher customer satisfaction scores (as shown by improved Net Promoter Scores) while maintaining stable employee sentiment. 4. 𝐋𝐨𝐰𝐞𝐫 𝐀𝐭𝐭𝐫𝐢𝐭𝐢𝐨𝐧 𝐑𝐚𝐭𝐞𝐬: Interestingly, the AI support led to reduced attrition rates, especially among new hires with less than six months of experience. 5. 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐞𝐝 𝐖𝐨𝐫𝐤𝐟𝐥𝐨𝐰𝐬: The AI system reduced the need for escalations to managers, improving vertical efficiency. However, its impact on horizontal workflows, like transfers between agents, showed mixed results, suggesting more refinement is needed in AI integration. 6. 𝐂𝐮𝐬𝐭𝐨𝐦𝐢𝐳𝐞𝐝 𝐀𝐈 𝐌𝐚𝐭𝐭𝐞𝐫𝐬: The software wasn’t off-the-shelf; it was a custom-built solution tailored to the company’s needs using the GPT family of language models. This emphasizes the importance of context-specific AI applications for effective outcomes. For leaders, managers, and AI practitioners, these insights are invaluable—highlighting not just the potential of AI, but also the nuanced ways it reshapes workflows, impacts employee dynamics, and transforms customer experiences.So, does generative AI really make a difference? According to this study, the answer is a resounding yes—but it depends on how thoughtfully it is deployed. Link 🔗 to the paper: https://lnkd.in/ejhUfufz

Just out: Quantifying the impact of #genAI on job performance, by Erik Brynjolfsson & team: "Access to AI assistance increases worker productivity, as measured by issues resolved per hour, by 15% on average, with substantial heterogeneity across workers. The effects vary significantly across different agents. Less experienced and lower-skilled workers improve both the speed and quality of their output, while the most experienced and highest-skilled workers see small gains in speed and small declines in quality. We also find evidence that AI assistance facilitates worker learning and improves English fluency, particularly among international agents. While AI systems improve with more training data, we find that the gains from AI adoption are largest for moderately rare problems, where human agents have less baseline experience but the system still has adequate training data. Finally, we provide evidence that AI assistance improves the experience of work along several dimensions: customers are more polite and less likely to ask to speak to a manager." Open access: https://lnkd.in/d4UecpnQ

Last week, I talked about the possibilities of AI to make work easier. This week, I want to share a clear example of how we are doing that at HubSpot. We’re focused on helping our customers grow. So naturally, we take customer support seriously. Whether it’s a product question or a business challenge, we want inquiries to be answered efficiently and thoughtfully. We knew AI could help, but we didn’t know quite what it would look like! We first deployed AI in website and support chat. To mitigate any growing pains, we had a customer rep standing by for questions that came through who could quickly take the baton if things went sideways. And, sometimes they did. But we didn’t panic. We listened, we improved, and we kept testing. The more data AI collects, the better it gets. Today, 83% of the chat on HubSpot’s website is AI-managed and our Chatbot is digitally resolving about 30% of incoming tickets. That’s an enormous gain in productivity! Our customer reps have more time to focus on complex, high touch questions. AI also helps us quickly identify trends—questions or issues that are being raised more frequently—so we can intervene early. In other words, AI has not just transformed our customer support. It has elevated it. So, here is what we learned: Don’t panic if customer experience gets worse initially! It will improve as your data evolves. Evolve your KPIs and how you measure success- if AI resolves typical questions and your team resolves tricky ones, they will need more time. Use AI to elevate your team's efforts How are you using AI in support? What are you learning?