Arize AI

Been working with more and more product managers in my day-to-day, and I can honestly say the PM role is changing significantly. It’s great to see PMs building AI products take their own AI journey, which usually looks something like this: 1 — a bit of uncertainty about where their role fits in the AI world, and the realization that they need to reinvent themselves 2 — educating themselves and getting familiar with using AI tools in their day-to-day (PRDs, specs, etc.), building a better understanding of concepts like RAG, MCP (tool calls), agent orchestration, and the different types of evals 3 — actually running evals, iterating, and experimenting alongside AI engineers to make AI systems better serve the business — becoming the bar for quality for their product and measuring the outcomes

Arize AX is listed as an Emerging Leader in the "Emerging Market Quadrant for Generative AI Engineering" in Gartner's latest "Innovation Guide for Generative AI Engineering" report (13 November)! Gartner customers can download the report here: https://lnkd.in/gPBWDG6u Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose. GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.

Microsoft’s red teaming agent in Microsoft Foundry generates sophisticated prompts designed to simulate adversarial attacks. Arize AX can help make these vulnerabilities visible and actionable. Used together, teams can create a complete workflow for self-improving agent security. This new blog + notebook by Richard Young walks through a practical example you can adapt to your use case: https://lnkd.in/grb6ge4k Here's the typical flow: ⛉ Run probes on your agent with AI red teaming agent in Microsoft Foundry ⛉ Arize AX captures traces and observability data from probes ⛉ Arize AX online evaluations flag regressions and provide explanation details ⛉ Send regressions to humans to annotate and create golden datasets ⛉ Feed the golden dataset to Arize AX prompt optimizer to iterate on prompt ⛉ Validate the performance of before and after prompts ⛉ Deploy the change and repeat the loop Again, our thanks to Sebastian Kohlmeier, Ilvens Jean, Andrew Tawaststjerna, Brittany Case, Bea Nallar, Elizabeth Fitzgerald, Joonseok Oh, Rohit Tatachar, and Chhavi Nijhawan and the Microsoft Azure team for their partnership!

Microsoft Foundry offers a single place to use any model — including both Claude and GPT frontier models — and framework, with enterprise controls to build AI apps and agents at scale. Coupled with Arize AI, developers can create create a continuous feedback system where the same evaluators that power offline testing also monitor live production traffic. As Sebastian Kohlmeier, Principal PM Manager for Foundry Observability at Microsoft notes: “Modern AI systems demand integrated visibility across development and production. It’s great to see Arize enabling seamless interoperability with Microsoft Foundry evals to power quality and safety evaluations.” In a new post and notebook, Richard Young walks through a concrete example of Microsoft Foundry + Arize AX for a ⚠️ content safety ⚠️ use case. Using Microsoft Azure content safety evaluators as an example, teams can see how the entire feedback loop functions, from trace export to dataset benchmarking to dashboard insight. ⛑️ Dive in: https://lnkd.in/g3_cEKwf Our thanks to Ilvens Jean, Andrew Tawaststjerna, Brittany Case, Bea Nallar, Elizabeth Fitzgerald, Joonseok Oh, Rohit Tatachar, Sebastian Kohlmeier, Chhavi Nijhawan and the teams at Microsoft AI, Microsoft Azure Communities and Microsoft Learn for their partnership!

Learn how to build better AI applications with this introduction to LLM-as-a-judge evaluation! Key takeaways: ➡️ Start by observing real application data to understand failure modes before defining metrics. ➡️ Keep metrics specific, answerable, and actionable - avoid catch-all "quality" scores or having 10+ overlapping metrics. ➡️ When writing judge prompts, treat them like human annotation guidelines: keep context under 4K tokens, and don't overthink fancy prompting techniques. ➡️ Most importantly, create a golden dataset with human annotations to measure judge alignment through meta-evaluation. Elizabeth Hutton emphasizes that imperfect evals beat no evals, and this is an iterative process with two overlapping loops - one improving your application, another improving your evaluations. She also covers when to use code vs human vs LLM evals, explains pairwise vs direct scoring approaches, and shares practical pitfalls to avoid. Check out the full video: https://lnkd.in/gZsYwRxK

We recently benchmarked Prompt Learning, Arize’s prompt-optimizer, against GEPA and saw that Prompt Learning matches (and often exceeds) GEPA’s accuracy in a fraction of the rollouts. Since launching Prompt Learning in July, the question we hear most is: “Prompt Learning or GEPA — which should I use?” To answer it, we re-created the full GEPA benchmark suite, measured rollout efficiency, and compared the end-to-end developer experience across both systems. The results: Prompt Learning achieves similar or better accuracy with far fewer rollouts, thanks to richer evaluation signals and trace-aware feedback loops. 🔁 Both systems share the same optimization loop run → evaluate → improve → repeat Both use meta-prompting and trace-level reflection so the optimizer learns from real application behavior, not static prompts. Under the hood, it’s essentially an RL-style feedback loop applied to prompts. 🔍 Where the gains came from GEPA brings powerful search machinery — evolutionary search, Pareto selection, prompt merging — but our tests showed that the largest improvements didn’t come from more search. They came from better evaluations. Evaluators that explain why an answer was wrong (not just that it was wrong) produced much stronger learning signals. Trace-aware evals (like hop-by-hop reasoning checks) helped Prompt Learning correct the exact failure mode instead of blindly exploring prompt space. TL;DR: Higher-quality evaluator prompts → faster (and sometimes stronger) optimization. Example evals here: https://lnkd.in/gPdbmYBj 🧩 Framework-agnostic by design Both GEPA and Prompt Learning support trace-level optimization, but GEPA requires your full application to be written in DSPy to enable tracing. Prompt Learning is framework-agnostic: LangChain, CrewAI, Mastra, AutoGen, vector DBs, custom stacks — anything. Add OpenInference tracing, export traces, and optimize. No lock-in. No rewrites. Start tracing your agents: https://lnkd.in/gznD_mAb 🛠️ No-code optimization & collaboration Prompt Learning also ships with a full no-code workflow inside Arize: - Run optimization experiments - Track iterations in the Prompt Hub - Test variants in the Prompt Playground Perfect for teams who want governance + collaboration without managing huge prompts directly in Git. If you want a deeper dive into the benchmarks and architectural differences, the full write-up is here: https://lnkd.in/gJxM3rxJ Try Prompt Learning: https://lnkd.in/gizYRBhN Open-source SDK: https://lnkd.in/g75cX3XB

Yongchao Chen with Harvard University and Massachusetts Institute of Technology covered the Google paper "TUMIX: Multi-Agent Test-Time Scaling with Tool-Use Mixture" at our latest community paper reading! Watch: https://lnkd.in/gn9yuQR6 The paper proposes Tool-Use Mixture (TUMIX), an ensemble framework that runs multiple agents in parallel, each employing distinct tool-use strategies and answer paths. Agents in TUMIX iteratively share and refine responses based on the question and previous answers. In experiments, TUMIX achieves significant gains over state-of-the-art tool-augmented and test-time scaling methods.

Arize is now part of Google's Gemini CLI extensions ecosystem! Our MCP Server Tracing Assistant's integration with Gemini CLI makes it easy to instrument AI applications with Arize AX. In your Gemini CLI, you can now ask natural-language questions like: 🟪  “Instrument this app using Arize AX.” 🟪  “Can you use manual instrumentation so that I have more control over my traces?” 🟪  “How can I redact sensitive information from my spans?” 🟪  “Can you make sure the context of this trace is propagated across these tool calls?” 🟪  “Where can I find my Arize keys?” 🔍 How to find: Go to the extensions tab of https://geminicli.com/ and type in 'arize'. Hat tip to Taylor Mullen, Raïssa Tona, Jack Wotherspoon, Erin Franz, Mukul Goyal, Google for Developers, Google Cloud and our own Richard Young and Noah Smolen for their support!

Google ADK’s strength lies in its code-first, modular approach to multi-agent orchestration. Rather than forcing rigid patterns, it offers Python flexibility with built-in support for state management, callbacks, and streaming. Together, ADK and Arize AX deliver a unique and unified experience for building, deploying, and refining agent systems. ADK orchestrates; Arize AX observes and optimizes. Through shared OpenTelemetry standards, ADK agents send telemetry directly to Arize AX without vendor lock-in -- and Arize AX delivers observability and evaluation trusted at a trillion-inference scale. In a new post, Richard Young walks through building a real travel concierge system that demonstrates both frameworks working together. You see how ADK's modular architecture enables complex agent coordination while the Arize AX platform provides visibility into every decision, tool call, and handoff. ✈️ Dive in: https://lnkd.in/gCT9Znbs

Here is Gemini CLI’s November 3rd weekly update for v0.15.0 - 🎉 Seamless Scrollable UI & Mouse Support: We’ve given the Gemini CLI a major facelift to make your terminal experience smoother and much more polished. You now get a flicker-free display with sticky headers that keep important context visible and a stable input prompt that doesn't jump around. We even added mouse support so you can click right where you need to type! (jacob314). - Announcement: https://lnkd.in/gFQHbW-n - 🎉 New Partner Extensions: - Arize: seamlessly instrument AI applications with Arize AX and grant direct access to Arize support gemini extensions install https://lnkd.in/gyZtiGG3 - Chronosphere: retrieve logs, metrics, traces, events, and specific entities gemini extensions install https://lnkd.in/gZYEtahT - Transmit: comprehensive context, validation, and automated fixes for creating production-ready authentication and identity workflows. gemini extensions install https://lnkd.in/gQGBfcgn - Todo Planning: Complex questions now get broken down into todo lists that the model can manage and check off. (anj-s) - Disable GitHub Extensions: Users can now prevent the installation and loading of extensions from GitHub. (kevinjwang1) - Extensions Restart: Users can now explicitly restart extensions using the /extensions restart command. (jakemac53) - Better Angular Support: Angular workflows should now be more seamless (MarkTechson) - Validate Command: Users can now check that local extensions are formatted correctly. (pr by @kevinjwang1) https://lnkd.in/gddtXp57 🧵