Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI & Milvus

1 | © Copyright 2025 Zilliz
1
Star Milvus
for a chance
to win a prize
tonight!

2
This just dropped (open-source edition)!
https://github.com/canopyai/Orpheus-TTS
https://huggingface.co/ds4sd/SmolDocling-256M-preview
https://huggingface.co/nvidia/GR00TN12B

3 Data Source: The Digitization of the World by IDC
10%
Other
of newly generated data in
2025 will be unstructured data
90%
The world is much more than just text and keywords

4
4
Mission:
Helping organizations make sense
of unstructured data.
2017
Founded
$113M
Raised
140
Employees
Redwood City, CA
Headquarters

5
5 | © Copyright 10/22/23 Zilliz
Welcome Speakers
From O1 to O3 How
Reasoning Transformed AI
and Launched a New
Generation of Agents
Building an Agent to Reason
about Private Data with
OpenAI & Milvus
TECH TALK 1 TECH TALK 2
Harry Zhao
Member of Technical Staff, OpenAI
Stefan Webb
Developer Advocate, Zilliz

6
6 | © Copyright 10/22/23 Zilliz
Stefan Webb
stefan.webb@zilliz.com
https://www.linkedin.com/in/stefan-webb
Building a Reasoning Agent with Milvus and OpenAI

7
01 Why Milvus?

8
8
Vector search
is the new
standard

9
A New tool emerged. The Vector Database

10
10
Milvus is an Open-Source Vector Database to
store, index, manage, and use the massive
number of embedding vectors generated by
deep neural networks and LLMs.
contributors
430
stars
33K
docker pulls
67M
forks
2.8K
+
Milvus: The most widely-adopted vector database

11
11
Zilliz was built by a top-tier team
of algorithm and database
engineers with a strong
pedigree in developing
high-performance, scalable,
and highly available distributed
systems, uniquely tailored for
vector search.
Built by
database & AI
experts

12
12
Milvus
Architecture
Fully distributed, designed for scalability
Design Principles
• Separation of storage and
compute
• Fully depend on mature storage
systems
• Microservice - scale by
functionality
• Separate streaming and
historical data
• Pluggable engine, storage and
index
• Log as data

Rich functionality
Bulk Import GPU, Intel & ARM
CPU support
Disk Based
Index
Tiered Storage
Million+ level
tenant support
Hybrid Search
Dense & Sparse
RBAC, TLS,
Encryption
Float, Binary, &
Sparse Vector
Tag+Vector
Optimized Filtering
Dynamic Schema

14
Use Case: Drug Discovery
Vectors: 12 Billion
Reqʼts: High Recall
Index: BIN_FLAT
Use Case: Data Search
Vectors: 2 Billion
Reqʼts: 200 ms, Cost mgmt
Index: DiskANN for cost savings
Use Case: Image Search
Vectors: 20 Billion
Reqʼts: High Insertion, Cost
Index: Disk Based Index
Use Case: Recommender System
Vectors: 20 Billion
Reqʼts: 5,000 QPS
Index: HNSW & CAGRA
Industry leaders already use vector search in their apps

15
Milvus Users

16
Set up Once: Common API across all products regardless of architecture
Flexible Deployment Options for Milvus

17
Framework
Hardware
Infrastructure
Embedding Models LLMs
Software Infrastructure
Vector Database
AI Stack

18
02 DeepSearcher demo

19
Research Agents
Iteration
“…learned to plan and
execute a multi-step
trajectory…”
“…backtracking and reacting
to real-time information…”
“…pivoting as needed in
reaction to information it
encounters…”
Search
“…trained using end-to-end
reinforcement learning on hard
browsing and reasoning tasks
across a range of domains…”
“…optimized for web browsing
and data analysis…”
Reasoning
“…fine-tuned on the upcoming
OpenAI o3 reasoning model…”
““…leverages reasoning to search,
interpret, and analyze massive
amounts of text…””

20
DeepSearcher from Zilliz

21

22
our query/task

23
initial subqueries

24
think what action to perform
search 1st subquery, 1st source

25
retrieve chunks with Milvus

26
search 1st subquery, 2nd source

27
think what action to perform
search 2nd subquery, 1st source

28

29

30
reflect on iter 1, new subqueries

31
search 1st subquery

32

33

34
search 2nd subquery

35

36

37
search 3rd subquery

38

39

40
reflect on iter 2,
no new subqueries

41
generate report

42

43

44
03 Letʼs build it ourselves!

45
🦺 🪚 👾
Itʼs Coding Time!
https://bit.ly/41HjZbJ
🖥 🛠 💻

46
04 Where to from here?

47
Where to from here?
Cost
● Specialized
hardware
● Algorithmic
improvements
Data
● Open-source
● High-quality
● Reasoning traces
Scaling
● Per-se

48
BRING YOUR OWN CLOUD
Zilliz BYOC
For Private VPCs
Milvus
Most widely-adopted open
source vector database
SELF MANAGED SOFTWARE
Zilliz Cloud
AI Powered Search that is
performant and scales
FULLY MANAGED SERVICE
Set up Once: Common API across all products regardless of architecture
Zilliz Offerings
Coming Soon!
https://cloud.zilliz.com/signup

49
https://milvus.io/discord
LET’S STAY CONNECTED!
Stefan Webb
Book a Free 11 Session
For Support Productionizing Milvus
http://bit.ly/43YkEYW

Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI & Milvus

More Related Content

Similar to Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI & Milvus

More from Zilliz

Recently uploaded

Hands-on Tutorial: Building an Agent to Reason about Private Data with OpenAI & Milvus