Tips for AI Experimentation and Controlled Deployment

DEPLOY AI AGENTS THE RIGHT WAY Over the past few years, I’ve watched teams and leaders race to deploy AI agents—chasing the latest LLM tools, spinning up proof-of-concepts, and hoping automation would “just work.” I made a lot of those mistakes myself. Looking back, I wish someone had handed me a blunt list of what actually matters when deploying AI agents in the real world. Here’s what I learned the hard way: If you start with technology instead of a real business problem, you’re setting yourself up for wasted effort. Everyone gets excited by the shiny stuff, but you only get real impact (and real wins) by picking a painful, high-value business problem and focusing relentlessly on solving that. Don’t trust your data “as-is.” No matter how confident you are, your data will need more cleaning, validation, and governance than you expect. It’s boring work, but skipping it will cost you months in rework and lost credibility. Involve stakeholders early—don’t treat AI agent deployment as a tech project only. If the business, end users, or compliance teams aren’t bought in, even the best agents will fail to gain traction. Automate what you can (retraining, monitoring, feedback), but never abdicate responsibility. “Set and forget” is a myth. Humans need to stay in the loop, especially when things go sideways or when continuous learning is needed. Version everything—models, data, code. It sounds trivial until something breaks and you can’t roll back or audit what changed. Align every metric to a business outcome. Technical wins are nice, but nobody outside the data team cares about incremental accuracy unless it moves the business needle—customer satisfaction, cost savings, regulatory wins. Document as you go. New teams will join, people will move on, and “tribal knowledge” fades fast. Documentation is how you scale and sustain real progress. Normalize sharing failures. It’s uncomfortable, but it’s how teams learn and avoid repeating mistakes. The fastest learning happens when people are open about what didn’t work. Watch out for risk and ethics. Bias, compliance, and privacy issues will creep in if you don’t proactively manage them. The cost of ignoring this is much higher down the road. Final point: Deploying AI agents isn’t “one and done.” Business needs and data drift, so build feedback and improvement into the process from day one. If you’re about to launch your first (or tenth) AI agent, keep it simple: Solve a real business pain. Get your data in shape. Keep the people loop tight. Share both your wins and your scars. #AILeadership #AIAgents #DigitalTransformation #EnterpriseAI #BusinessStrategy

⚠️ Every untested prompt is a potential bottleneck in your AI's performance! Behind every prompt lies a system of expectations that must be tested, measured, and refined like any other engineering artifact. I've been watching teams discover that the same prompt can trigger spam filters for enterprise clients while working perfectly for startups. I have 57 other examples (couple of them captured below)! The difference between a prompt that scales and one that fails isn't creativity - it's systematic evaluation. The most successful AI implementations I'm seeing now treat prompt engineering like software development: test, measure, version control, deploy and thats what we have streamlined and automated Future AGI by integrating our evals right into the prompt workbench. Here is how it is helping teams of all sizes-: 1. Parallel Testing at Scale- While we're all running A/B tests one at a time, this runs thousands of prompt variations simultaneously. Think: 38% accuracy jumping to 75% - but now you know exactly which prompt got you there. 2. Beyond Gut Feelings- Custom metrics that actually matter. Not just "does it work?" but "how does it perform against YOUR specific success criteria?" 3. Side-by-Side Reality Check- Every variation laid out visually. No more spreadsheet hell or manual tracking. The winning patterns become obvious. 4. Production-Ready Deployment- Version control built-in. Test → Validate → Ship. One commit at a time. As quoted by GPT- you can’t build a skyscraper with LEGO instructions and so is the case with your AI, you cant build it on fragile prompts. What are your biggest prompt engineering challenges? #AIEngineering #PromptEngineering #ProductDevelopment #GenAI #MachineLearning

Some of the best AI breakthroughs we’ve seen came from small, focused teams working hands-on, with structured inputs and the right prompting. Here’s how we help clients unlock AI value in days, not months: 1. Start with a small, cross-functional team (4–8 people) 1–2 subject matter experts (e.g., supply chain, claims, marketing ops) 1–2 technical leads (e.g., SWE, data scientist, architect) 1 facilitator to guide, capture, and translate ideas Optional: an AI strategist or business sponsor 2. Context before prompting - Capture SME and tech lead deep dives (recorded and transcribed) - Pull in recent internal reports, KPIs, dashboards, and documentation - Enrich with external context using Deep Research tools: Use OpenAI’s Deep Research (ChatGPT Pro) to scan for relevant AI use cases, competitor moves, innovation trends, and regulatory updates. Summarize into structured bullets that can prime your AI. This is context engineering: assembling high-signal input before prompting. 3. Prompt strategically, not just creatively Prompts that work well in this format: - “Based on this context [paste or refer to doc], generate 100 AI use cases tailored to [company/industry/problem].” - “Score each idea by ROI, implementation time, required team size, and impact breadth.” - “Cluster the ideas into strategic themes (e.g., cost savings, customer experience, risk reduction).” - “Give a 5-step execution plan for the top 5. What’s missing from these plans?” - “Now 10x the ambition: what would a moonshot version of each idea look like?” Bonus tip: Prompt like a strategist (not just a user) Start with a scrappy idea, then ask AI to structure it: - “Rewrite the following as a detailed, high-quality prompt with role, inputs, structure, and output format... I want ideas to improve our supplier onboarding process with AI. Prioritize fast wins.” AI returns something like: “You are an enterprise AI strategist. Based on our internal context [insert], generate 50 AI-driven improvements for supplier onboarding. Prioritize for speed to deploy, measurable ROI, and ease of integration. Present as a ranked table with 3-line summaries, scoring by [criteria].” Now tune that prompt; add industry nuances, internal systems, customer data, or constraints. 4. Real examples we’ve seen work: - Logistics: AI predicts port congestion and auto-adjusts shipping routes - Retail: Forecasting model helps merchandisers optimize promo mix by store cluster 5. Use tools built for context-aware prompting - Use Custom GPTs or Claude’s file-upload capability - Store transcripts and research in Notion, Airtable, or similar - Build lightweight RAG pipelines (if technical support is available) - Small teams. Deep context. Structured prompting. Fast outcomes. This layered technique has been tested by some of the best in the field, including a few sharp voices worth following, including Allie K. Miller!

Don't be afraid of hallucinations! It's usually an early question in most talks I give on GenAI "But doesn't in hallucinate? How do you use a technology that makes things up?". It's a real issue, but it's a manageable one. 1. Decide what level of accuracy you really need in your GenAI application. For many applications it just needs to be better than a human, or good enough for a human first draft. It may not need to be perfect. 2. Control your inputs. If you do your "context engineering" well, you can point it to the data you want better. Well written prompts will also reduce the need for unwanted creativity! 3. Pick a "temperature". You can select a model setting that is more "creative" or one that sticks more narrowly to the facts. This adjusts the internal probabilities. The "higher temperature" results can often be more human-like and more interesting. 4. Cite your sources. RAG and other approaches allow you to be transparent about what the answers are based on, to give a degree of comfort to the user. 5. AI in the loop. You can build an AI "checker" to assess the quality of the output 6. Human in the loop. You aren't going to just rely on the AI checker of course! In the course of a few months we've seen concern around hallucinations go from a "show stopper" to a "technical parameter to be managed" for many business applications. It's by no means a fully solved problem, but we are highly encouraged by the pace of progress. #mckinseydigital #quantumblack #generativeai

"So what do we need while building an AI application?" At Vellum, we've spoken to over 1,500 people at varying maturities of using LLMs in production. It’s easy to whip together a prototype of an AI-powered feature using popular open source frameworks, but we repeatedly see people having difficulty crossing the prototype-to-production chasm. They deploy to production and then quickly run into countless edge cases and wonder why their AI application isn’t working well. 4 key building blocks emerge in all successful use cases 👇 1: Data An LLM is trained on the general internet and doesn't have access to your data. An LLM is also inherently stateless. The Data pillar is all about providing the right context to your prompt at run-time. Some questions to consider: - What data is unique to you? - How do you best structure and query this data at run-time? This may include experimentation on your RAG pipelines and memory management - Which prompt/model to use to get a meaningful output? 2: Experimentation Unlike traditional software engineering, LLMs are non-deterministic and require a lot of trial and error to get right. Here's what you should keep in mind: - Before starting anything, what's the right architecture for my application? Single prompt or chains? RAG? API calls? - What are the right eval metrics to test improvements in my experiments? - How many test cases should I use? - Who will do the testing and experimentation? Can it be offloaded to non technical team members? 3: Lifecycle management We're used to having good tools for software engineering: Datadog for monitoring, GitHub for version control and CircleCI for CI/CD. But none of this exists for LLMs. Without this tooling you're kinda flying blind. Here's what to do: - Maintain detailed logs, for prompts and prompt chains - Use charts and set up alerting - Capture user feedback on completions - Keep a separate staging and production version of your app - Replay historical requests while making changes to prevent regressions 4: Continuous improvement Now that you have all this data collected in step 3, use this to further strengthen your data moat (step 1) - Any edge cases in production should be added to your test bank - If you use dynamic few shot prompting via RAG, keep adding production data to your DB (a whole article on this to follow later) - Build a caching layer to save cost - Fine tune a model to lower costs if your task is standardized This may seem like a lot to build, but luckily, you don't have to build it yourself. Message me if you'd like a demo of Vellum. More details about these pillars in comments

I love using Anthropic's Claude as a thought partner. Recently, "we" have been having weekly reflection sessions. I feed the 5 most interesting meeting transcripts from my past week into Claude's enormous context window and ask it to read them closely and ask me 3–5 hard-hitting follow-up questions. Then, I answer those questions with a long-form monologue of 45–90 minutes in length while driving or on a walk, pop the transcript back into Claude, and continue until we've gotten polished nuggets of insight. Here's what "we" learned this week after our latest reflection session, based on my work helping clients get their teams focused with OKRs and accelerate progress toward their goals with AI: 1. Automation enables augmentation. By delegating repetitive tasks to AI, you free up human attention/cognition to focus on higher-value efforts and innovation. 2. Integration with existing foundation models (GPT-4, etc.) is preferable to building custom AI initially in most cases. Leverage their scale while focusing your differentiation elsewhere. 3. Tight feedback loops and manual problem-solving early on is critical to ensure you are solving actual (vs. assumed) problems and properly designing solutions. No shortcuts. 4. Focus first on amplifying people’s strengths by delegating tasks they have to do but may not enjoy vs. attempting to outright replace roles. More humane and effective. 5. Specialized, proprietary data remains a key competitive advantage that should integrate with commoditized models. I highly recommend this exercise! It can help you make sense of lots of data in a short period of time, while remaining focused on the stuff that's most interesting, valuable, and deserving of your your most important resource—your attention.

This week I presented to a large CRE brokerage & prop mgmt firm on how to leverage AI in their business: Here’s the key takeaways: 1. AI is fundamentally changing the way that people access information. It’s a powerful tool for you to do market research, it’s just as powerful for your clients. For better or worse value of your market reports that clients find on Google is getting slowly deleted. 2. Your ability to create content that tells a story, cuts thru the noise and proactively gets in front of your target audience is at an all time high if you can leverage a handful of AI tools. 3. What AI can do for you depends entirely on who you are. A company with 1000+ brokers & employees has entirely different needs and pain points than a 20 person shop, or a solo operator. The solo operator will have the most to gain quickly, the larger you are, the more thoughtful you’ll need to be about what you can do as an organization to add value in the age of AI. 3.A. This extends to individuals as well as companies. A high producing broker should leverage AI to minimize minutia so they can do more deals. A junior broker probably shouldn’t, at least not at first. The minutia is how they learn to be a broker. There are plenty of other ways to leverage AI to do MORE, not just do less. 4. The larger your firm, the more quality control becomes important. Yes, you can now take an image of a property and morph it into an aerial video with AI. Do you want 1,000 individual brokers doing this independently? Probably not. AI can still make mistakes. AI enables you to do more, but you’ll want to extend your existing quality control to however you begin leveraging AI, especially with marketing. 5. There’s 3 basic levels to leveraging AI. 1: Test and experiment - find the tools you think will actually save you time, and test them to see if they actually do. 2: Customize AI to work for you - If you’re getting ChatGPT or another LLM to execute a specific task consistently, you can likely create a customized version so every time you open up that GPT, it already knows what you want it to do, saving you even more time. 3: Automate - If you were able to customize AI to execute a task, there’s a good chance you can set it up to execute the task automatically, so you never even need to open up AI. 6. Automation isn’t the click of a button. There’s boring work to do first, you need a clear outline of how your process works now BEFORE you automate it with AI. You’ll have some decisions to make about how the process works now that you’re automating it with AI, and all who want to benefit from it will need to be on the same page. If the process isn’t written down, it doesn’t exist, and you can’t automate a process that doesn’t exist. 7. How do you stay ahead of the curve when AI is moving so fast? Simple answer: Learn to use it. Pick the handful of tools you think will be helpful to you and your business, put them to work.

𝗨𝗻𝗰𝗵𝗮𝗿𝘁𝗲𝗱 𝗧𝗲𝗿𝗿𝗶𝘁𝗼𝗿𝘆: 𝗪𝗵𝘆 𝘁𝗵𝗲 𝗡𝗲𝘅𝘁 𝗧𝗵𝗿𝗲𝗲 𝗬𝗲𝗮𝗿𝘀 𝗪𝗶𝗹𝗹 𝗥𝗲𝘄𝗿𝗶𝘁𝗲 𝘁𝗵𝗲 𝗥𝘂𝗹𝗲𝘀 Artificial Intelligence is reshaping business, work, and daily life at an unprecedented pace. Here's how to navigate the coming changes: 𝗧𝗵𝗲 𝗦𝗰𝗮𝗹𝗲 𝗪𝗲'𝘃𝗲 𝗡𝗲𝘃𝗲𝗿 𝗦𝗲𝗲𝗻 📈 Generative AI usage has erupted from novelty to necessity in just 30 months. 📈 OpenAI processes 2.5 billion prompts daily or 29,000 every second. 📈 ChatGPT's mobile-only user base now exceeds 540 million monthly active users, and is still climbing. 📈 No previous technology (electricity, internet, smartphones) scaled this quickly. And, AI is improving daily making it even more useful while even more users adopt it. 𝗧𝗵𝗲 𝗩𝗲𝗹𝗼𝗰𝗶𝘁𝘆 𝗣𝗿𝗼𝗯𝗹𝗲𝗺 🚀 Adoption: Each new AI release spreads instantly, compounding usage worldwide. 🚀 Capability: Today's "wow" demo feels quaint after just six weeks. 🚀 Measurement: What counts more—prompts, tokens, agents, or the invisible tasks completed overnight? 🚀 Anyone predicting three years out is either guessing or selling. But the near-term picture is sharpening as we get closer, and it's transformative. 𝗙𝗶𝘃𝗲 𝗠𝗼𝘃𝗲𝘀 𝘁𝗼 𝗠𝗮𝗸𝗲 𝗡𝗼𝘄 1. Run a Personal "Task Audit"   - Track everything you do for a week.   - Identify text, code, image, language, or data-heavy tasks.   - These are prime candidates for AI delegation. 2. Instrument Your Workflows   - Capture time saved, error rates, and decision latency.   - You can't prove improvement without measurement. - Business leaders want to hear about AI benefits in business language 3. Build an "AI Fluency Hour"   - Block 60 minutes a week for team experiments.   - Share both wins and failures openly.   - Shared fluency and best practices beats siloed "centers of excellence." 4. Draft Lightweight Guardrail Policies   - Don't wait for perfect governance.   - Set three clear red lines (e.g., no confidential data in AI, require human review for client work, always attribute sources).   - Update monthly as you learn. 5. Bet on Optionality   - Stay flexible: capital, tools, vendors, even models.   - In uncertainty, adaptation is your best asset. 𝗕𝘂𝗰𝗸𝗹𝗲 𝗨𝗽 The terrain will keep shifting, but progress won't wait. Those who learn and act now will have the best chances to compete and thrive in the expanding age of intelligence. I feel incredibly fortunate to be alive in this moment, as universal access to intelligence changes everything across business, careers, and society itself. 🚀 #AI #DigitalTransformation #Leadership #FutureOfWork