Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & Milvus

1 | © Copyright 8/16/23 Zilliz
Agentic AI in Action
Real-Time Vision, Memory & Autonomy with Browser Use &
Milvus
Webinar - 20250313
Stephen Batifol

Stephen Batifol
Developer Advocate, Zilliz / Milvus
About Me
stephen.batifol@zilliz.com
linkedin.com/in/stephen-batifol/
@stephenbtl

| © Copyright 8/16/23 Zilliz
3
What are Agentic
Systems?

● Workflows – Systems where LLMs and tools are orchestrated
through predefined code paths
● Agents – Systems where LLMs dynamically direct their own
processes and tool usage, maintaining control over how they
accomplish tasks.
Agentic Systems
https://www.anthropic.com/engineering/building-effective-agents

● Agentic Systems trade latency and cost for better performance
● Workflows offer predictability and consistency for well-defined
tasks
● Agents are a better option when flexibility and model-driven
decision-making are needed at scale.
When (and when not) to use Agents

Examples

Augmented LLM

Workflow: Prompt Chaining

Workflow: Routing

Agents

When to use Agents?
● Used for open-ended problems
○ Difficult to predict the required number of steps
○ Canʼt hardcode a fixed path
● Use sandboxed environments if possible
● Appropriate Guardrails

Vector Search
& Vector DBs

Vectors Unlock Unstructured Data

Vector Space

How Similarity Search works
Vn, 1
…
…
…
1
2
3
4
5
Transform into
Vectors
Unstructured Data
Images
User Generated
Content
Video
Documents
Audio
Vector Embeddings
Perform Approximate
Nearest Neighbor
Similarity Search
Perform Query
Get Results
Store in Vector Database

16
Milvus

● pip install on your laptop
● Plug into your favorite AI dev tools
● Push to production with a single line of code
Easy to start

Bulk Import GPU, Intel & ARM
CPU support
Disk Based
Index
Tiered Storage
Million+ level
tenant support
Hybrid Search
Dense & Sparse
RBAC, TLS,
Encryption
Float, Binary, &
Sparse Vector
Tag+Vector
Optimized Filtering
Dynamic Schema
Feature rich

Milvus Lite Milvus Standalone Milvus Distributed
● Ideal for prototyping,
small scale
experiments.
● Easy to set up and
use, pip install
pymilvus
● Scale to ≈1M vectors
● Run on K8s
● Load balancer and
Multi-Node
Management
● Scaling of each
component
independently
● Scale to 100B
vectors
● Single-Node
Deployment
● Bundled in a single
Docker Image
● Supports Primary/
Secondary
● Scale up to 100M
vectors
Ready to scale 🚀
Write your code once, and run it everywhere, at scale!
● API and SDK are the same

More than Vectors?

Future of search is combining different search techniques
1. Semantic Search
2. Keyword Search
3. Filtering
All, in one unified platform
Our Vision is more than just Vectors

● Vector search is great for semantic
understanding
○ Can miss exact keyword matches
● Run two separate systems
○ Vector DB for semantic search
○ Elasticsearch/similar for keyword search
Results in complex architecture and
operational overhead
The Search Dilemma

23
Milvus - Full Text Search

Why Full Text Search?
● Augment the Search Quality of Embedding based
Semantic Search
● Provide Search with more emphasis on Keyword
Matching
● Easy Hybrid Search of BM25  Dense
Embeddings in a single system.

Milvusʼ Approach
The user can almost forget about Vectors
● Insert the raw text into Milvus and search using the text.
Milvus takes care of
● Text Analyzing and Tokenization
● Term distribution statistics management
● Document/Query vector encoding
● BM25 based scoring

Full Text Search

What we are building

A Search on X about Milvus

● Combining visual understanding with context awareness
● Build an assistant that knows the difference between a black kiteʼs
migration patterns and a new article about Milvus Vector DB
● In the future - Tell us about what Users are saying
AI for Smarter Browsing on Socials

Tech Stack

● Enable AI Agents to control your browser
Features:
● Multi-tab Management
● Vision + HTML Extraction
● Custom Actions
● Self-Correcting
● Different LLM support
Browser Use

● Natively multimodal
● Strong performance on multimodal tasks,
excels in instruction following
● 1M Input token
● Supports Text, Image, Video, Audio
Gemini Flash

Structured Output
Make it possible to generate either a JSON or a Pydantic object.
Benefits:
● Reliable type-safety: No need to validate or retry incorrectly
formatted responses
● Explicit refusals: Safety-based model refusals are now
programmatically detectable
● Simpler prompting: No need for strongly worded prompts to
achieve consistent formatting

Structured Output
JSON Pydantic

● pip install on your laptop
● Plug into your favorite AI dev tools
● Push to production with a single line of code
Milvus

Architecture

37
Demo!

milvus.io
github.com/milvus-io/
@milvusio
@stephenbtl
/in/stephen-batifol
Thank you

Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & Milvus

More Related Content

What's hot

Similar to Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & Milvus

More from Zilliz

Recently uploaded

Agentic AI in Action: Real-Time Vision, Memory & Autonomy with Browser Use & Milvus