Optimizing AI Email Agent Performance

I discovered I was designing my AI tools backwards. Here’s an example. This was my newsletter processing chain : reading emails, calling a newsletter processor, extracting companies, & then adding them to the CRM. This involved four different steps, costing $3.69 for every thousand newsletters processed. Before: Newsletter Processing Chain (first image) Then I created a unified newsletter tool which combined everything using the Google Agent Development Kit, Google’s framework for building production grade AI agent tools : (second image) Why is the unified newsletter tool more complicated? It includes multiple actions in a single interface (process, search, extract, validate), implements state management that tracks usage patterns & caches results, has rate limiting built in, & produces structured JSON outputs with metadata instead of plain text. But here’s the counterintuitive part : despite being more complex internally, the unified tool is simpler for the LLM to use because it provides consistent, structured outputs that are easier to parse, even though those outputs are longer. To understand the impact, we ran tests of 30 iterations per test scenario. The results show the impact of the new architecture : (third image) We were able to reduce tokens by 41% (p=0.01, statistically significant), which translated linearly into cost savings. The success rate improved by 8% (p=0.03), & we were able to hit the cache 30% of the time, which is another cost savings. While individual tools produced shorter, “cleaner” responses, they forced the LLM to work harder parsing inconsistent formats. Structured, comprehensive outputs from unified tools enabled more efficient LLM processing, despite being longer. My workflow relied on dozens of specialized Ruby tools for email, research, & task management. Each tool had its own interface, error handling, & output format. By rolling them up into meta tools, the ultimate performance is better, & there’s tremendous cost savings. You can find the complete architecture on GitHub.

You've built your AI agent... but how do you know it's not failing silently in production? Building AI agents is only the beginning. If you’re thinking of shipping agents into production without a solid evaluation loop, you’re setting yourself up for silent failures, wasted compute, and eventully broken trust. Here’s how to make your AI agents production-ready with a clear, actionable evaluation framework: 𝟭. 𝗜𝗻𝘀𝘁𝗿𝘂𝗺𝗲𝗻𝘁 𝘁𝗵𝗲 𝗥𝗼𝘂𝘁𝗲𝗿 The router is your agent’s control center. Make sure you’re logging: - Function Selection: Which skill or tool did it choose? Was it the right one for the input? - Parameter Extraction: Did it extract the correct arguments? Were they formatted and passed correctly? ✅ Action: Add logs and traces to every routing decision. Measure correctness on real queries, not just happy paths. 𝟮. 𝗠𝗼𝗻𝗶𝘁𝗼𝗿 𝘁𝗵𝗲 𝗦𝗸𝗶𝗹𝗹𝘀 These are your execution blocks; API calls, RAG pipelines, code snippets, etc. You need to track: - Task Execution: Did the function run successfully? - Output Validity: Was the result accurate, complete, and usable? ✅ Action: Wrap skills with validation checks. Add fallback logic if a skill returns an invalid or incomplete response. 𝟯. 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗲 𝘁𝗵𝗲 𝗣𝗮𝘁𝗵 This is where most agents break down in production: taking too many steps or producing inconsistent outcomes. Track: - Step Count: How many hops did it take to get to a result? - Behavior Consistency: Does the agent respond the same way to similar inputs? ✅ Action: Set thresholds for max steps per query. Create dashboards to visualize behavior drift over time. 𝟰. 𝗗𝗲𝗳𝗶𝗻𝗲 𝗦𝘂𝗰𝗰𝗲𝘀𝘀 𝗠𝗲𝘁𝗿𝗶𝗰𝘀 𝗧𝗵𝗮𝘁 𝗠𝗮𝘁𝘁𝗲𝗿 Don’t just measure token count or latency. Tie success to outcomes. Examples: - Was the support ticket resolved? - Did the agent generate correct code? - Was the user satisfied? ✅ Action: Align evaluation metrics with real business KPIs. Share them with product and ops teams. Make it measurable. Make it observable. Make it reliable. That’s how enterprises scale AI agents. Easier said than done.

It's more important you feed in the right info than try nail the perfect prompt. The way businesses build with AI is changing. The shift is happening due to reasoning development & agentic capability (AI is able to figure out how to achieve an objective on its own). That means managing what information your AI agent has access to at any given moment is more important than tweaking prompts. You might think that giving AI agents access to everything would make them smarter. The opposite is true. As you add more info, AI performance declines aka "context rot." So here's what you need to do: - Keep instructions clear, no duplication - Don't overload your AI with complex rules - Give your AI just enough direction without micromanaging - Provide a focused toolkit where each function has a clear purpose, so 1 agent for each function rather than trying to get one agent to do everything - Let AI agents retrieve information on-demand For work that spans hours / days, use 2 approaches: 1. Summarizing conversation history to preserve what matters 2. Give agents the ability to take & reference their own notes The most effective AI deployments treat information as a strategic resource, not an unlimited commodity. Getting this right means faster, more reliable results from your AI investments. Image from Anthropic describing the evolution of prompt engineering into context engineering below. p.s. I'm looking to take on no more than 2 clients who want to build this layer into their business as part of a new framework I'm developing - focus is on B2B marketing.

Anthropic released an excellent article on building effective AI agents, and it has some great recommendations. Here’s what I’d add: (TL;DR - use the right frameworks that focus on simplicity and smart memory management) Their recommendations echoed a lot of what we’re seeing from our customers at Zep AI (YC W24): 1. Start simple: Use the simplest solution possible, only increasing complexity when needed. Many applications don't require full agents - a single well-optimized LLM call with smart memory retrieval can be enough. 2. Understand the taxonomy: Anthropic distinguishes between workflows (predefined code paths) and agents (systems where LLMs dynamically direct their own processes). Different problems need different approaches. 3. Use proven patterns: The most effective implementations use: - Prompt chaining (sequential LLM calls for accuracy) - Routing (directing inputs to specialized tasks) - Parallelization (simultaneous processing for speed or confidence) - Orchestrator-workers (dynamic task delegation) - Evaluator-optimizer (iterative refinement with feedback) 4. Pay attention to tool design: Treat "agent-computer interfaces" with the same care as human interfaces. Well-documented, intuitive tools are crucial for agent success. This is all excellent advice. I’d add two things: 1. Double clicking on tool advice - I’ve talked about this before, but giving the LLM access to the right tools, not every tool, is critical for performance. Too many tools results in poorer performance. 2. Memory layer is crucial - As these systems evolve, productionize, and integrate more and more features and data, stuffing all that information into the context windows gets less and less effective. Invest in a robust memory layer so that you’re providing the agent with the right knowledge at the right time, not all of it all the time. What would you add? What are you seeing in the enterprise?