Setting up Text Classifier for Support Emails

Recently, a client reached out to us expressing frustration with the RAG (Retrieval-Augmented Generation) application they had implemented for customer support emails by a different AI agency. Despite high hopes of increased efficiency, they were facing some significant problems: The RAG model frequently provided wrong answers by pulling information from the wrong types of emails. For example, it would respond to a refund request email with details about changing an order - simply because those emails contained some similar wording. Instead of properly classifying the emails by type and intent, it seemed to just perform a broad embedding search across all emails. This created a confusing mess where customers were receiving completely irrelevant and nonsensical responses to their inquiries. Rather than streamlining operations, the RAG implementation was actually making customer service much worse and more time-consuming for agents. The client's team had tried tuning the model parameters and changing the training data, but couldn't get the RAG application to accurately distinguish between different contexts and email types. They asked us to take a look and help get their system operating reliably. After analyzing their setup, we identified a few key issues that were derailing the RAG performance: Lack of dedicated email type classification The RAG model needed an initial step to explicitly classify the email into categories like refund, order change, technical support, etc. This intent signal could then better focus the retrieval and generation steps. Noisy, inconsistent training data The client's original training set contained a mix of incomplete email threads, mislabeled samples, and inconsistent formats. This made it very difficult for the model to learn canonical patterns. Retrieval without context filtering The retrieval stage wasn't incorporating any context about the classified email type to filter and rank relevant information sources. It simply did a broad embedding search. To address these problems, we took the following steps with the client: Implemented a new hierarchical classification model to categorize emails before passing them to the RAG pipeline Cleaned and expanded the training data based on properly labeled, coherent email conversations Added filtered retrieval based on the email type classification signal Performed further finetuning rounds with the augmented training set After deploying this updated system, we saw an immediate improvement in the RAG application's response quality and relevance. Customers finally started getting on-point information addressing their specific requests and issues. The client's support team also reported a significant boost in productivity. With accurate, contextual draft responses provided by the RAG model, they could better focus on personalizing and clarifying the text - not starting responses completely from scratch.

🧵 Real Stories of Generative AI in Action (Feature 6 of a multi-part series, you can access the full series at #AWSGenAIinAction) 🗣️ For all organizations with a customer service function, efficient classification and routing of queries is critical to ensure customers receive a speedy and accurate service experience. Recently, the AWS Generative AI Innovation Center (#GenAIIC) collaborated with leading property and casualty insurance carrier Travelers to address this challenge. 🏷️ Classify to accelerate: Travelers receives millions of emails a year with agent or customer requests to service policies, with 25% of emails containing attachments (e.g. ACORD insurance forms as PDFs). Requests involve areas like address changes, coverage adjustments, payroll updates, or exposure changes. The main challenge was classifying emails received by Travelers into service request categories. 👩💻 Designing an efficient system: To achieve the optimal balance of cost and accuracy, the solution employed prompt engineering on a pre-trained Foundation Model (FM) with few-shot prompting to predict the class of an email, all built on Amazon #Bedrock using Anthropic #Claude models. The teams manually analyzed 4K+ email texts and consulted with business experts to understand the differences between categories to provide sufficient explanations for the FM, including explicit instructions on how to classify an email. Additional instructions showed the model how to identify key phrases that help distinguish an email’s class from the others. The workflow starts with an email, then, given the email’s text and any PDF attachments, the email is given a classification of 13 defined classes by the model. ✅ Getting it right, and saving time: The Travelers solution (with prompt engineering, category condensing, document processing adjustments, and improved instructions) yielded classification accuracy to 91%, a 23 pt improvement vs. the original solution with just a pre-trained FM. Using the predictive capabilities of FMs to classify complex, and sometimes ambiguous, service request emails, the system will save tens of thousands of hours of manual processing and allow Travelers to redirect that time toward more complex tasks. 🌟 What I find most inspiring about this project is how it demonstrates the practical application of #GenAI to augment human capabilities. It's a great example of how organizations can use technology to optimize operations while maintaining focus on customer experience. Read more here: https://lnkd.in/et3P5XYp #AWS #CustomerService #Innovation #DigitalTransformation #Insurance #TechInnovation #AmazonBedrock

Most people set up email automation backwards. They start with responses instead of understanding. Here's the right way to build an email autoresponder that actually works: Prerequisites you need: - Gmail or similar email platform - OpenAI API access - Basic automation tool (Zapier works) Step 1: Build the Classifier Create categories that match your workflow: 1. Client inquiries (highest priority) 2. Important business matters 3. Informational updates 4. Promotional content 5. Urgent requests Step 2: Train the AI Feed your historical emails to teach the system YOUR communication patterns. Step 3: Set Up Smart Routing Different email types = different workflows: → Urgent emails: Instant notifications → Client emails: Priority queue + personal touch → Info emails: Auto-file and weekly digest Step 4: Test and Refine Start with 1 email category. Perfect it. Then expand. Common pitfall to avoid: Don't automate everything at once. Start small, learn, then scale.