Vector Search Innovations in Generative AI

We have seen recently a surge in vector databases in this era of generative AI. The idea behind vector databases is to index the data with vectors that relate to that data. Hierarchical Navigable Small World (HNSW) is one of the most efficient ways to build indexes for vector databases. The idea is to build a similarity graph and traverse that graph to find the nodes that are the closest to a query vector. Navigable Small World (NSW) is a process to build efficient graphs for search. We build a graph by adding vectors one after the others and connecting each new node to the most similar neighbors. When building the graph, we need to decide on a metric for similarity such that the search is optimized for the specific metric used to query items. Initially, when adding nodes, the density is low and the edges will tend to capture nodes that are far apart in similarity. Little by little, the density increases and the edges start to be shorter and shorter. As a consequence the graph is composed of long edges that allow us to traverse long distances in the graph, and short edges that capture closer neighbors. Because of it, we can quickly traverse the graph from one side to the other and look for nodes at a specific location in the vector space. When we want to find the nearest neighbor to a query vector, we initiate the search by starting at one node (i.e. node A in that case). Among its neighbors (D, G, C), we look for the closest node to the query (D). We iterate over that process until there are no closer neighbors to the query. Once we cannot move anymore, we found a close neighbor to the query. The search is approximate and the found node may not be the closest as the algorithm may be stuck in a local minima. The problem with NSW, is we spend a lot of iterations traversing the graph to arrive at the right node. The idea for Hierarchical Navigable Small World is to build multiple graph layers where each layer is less dense compared to the next. Each layer represents the same vector space, but not all vectors are added to the graph. Basically, we include a node in the graph at layer L with a probability P(L). We include all the nodes in the final layer (if we have N layers, we have P(N) = 1) and the probability gets smaller as we get toward the first layers. We have a higher chance of including a node in the following layer and we have P(L) < P(L + 1). The first layer allows us to traverse longer distances at each iteration where in the last layer, each iteration will tend to capture shorter distances. When we search for a node, we start first in layer 1 and go to the next layer if the NSW algorithm finds the closest neighbor in that layer. This allows us to find the approximate nearest neighbor in less iterations in average. ---- Find more similar content in my newsletter: TheAiEdge.io Next ML engineering Masterclass starting July 29th: MasterClass.TheAiEdge.io #machinelearning #datascience #artificialintelligence

Thrilled to share that vector search in Azure Cognitive Search is now in public preview. We first talked about the vector search capability at Build in May, and since then lots of developers signed up for the private preview and gave us lots of feedback. Very much looking forward to hearing more from everyone that wants to give this public preview a try. Vector representations have quickly improved over the last few years in many ways, from better quality to more versatility supporting different media types and multi-modality (e.g. text + images, text + sound). Using vector search we can put these vector embeddings to work to create great search experiences. Vector search also plays an important role in Generative AI applications that use the retrieval-augmented generation (RAG) pattern. The quality of the retrieval system is critical to these app’s ability to ground responses on specific data coming from a knowledge base. Not only Azure Cognitive Search can now be used as a pure vector database for these scenarios, but it can also be used for hybrid retrieval, delivering the best of vector and text search, and you can even throw-in a reranking step for even better quality by enabling it. Read more in the official announcement: https://lnkd.in/gEv3Xm3W To see all the details take a look at the vector search documentation: https://lnkd.in/gMRQfJjV We also updated the RAG demo GitHub repo to let you experiment with vector search and hybrid search in addition to existing retrieval modes: https://lnkd.in/gWEEJ6SN

Enterprises are struggling to build production-ready RAG systems with the #embedding-model + #vector-database + #LLM approach — and it’s because these current approaches are a pretty hacky way to build a system. The only way to fight low accuracy with these hacky approaches is by continuously fine-tuning the models, which requires customizing the system, model, and design by narrower and narrower use cases. This ultimately destroys the promise of "generative" AI. Luckily, the AI research community has developed several more robust techniques for retrieving data for question & answer use cases, beyond embeddings + vector db: #Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (https://lnkd.in/enfb5tBX): This paper discusses a system that empowers the LLM itself with the capacity to access external documents. This content is then used to formulate new and meaningful responses. The paper proves that RAG is a versatile tool for various NLP tasks like text summarization, question answering, etc. #Fusion-in-Decoder (https://lnkd.in/ejd4JBvh): FiD is a model which improves response generation through the direct use of passage retrieval properties. Unlike the traditional approach where the document-level probability is used only to rank documents, FiD employs a sequence-to-sequence model which measures the likelihood of a response given each source document separately and then only combines these likelihoods. #ColBERT (https://lnkd.in/eZ2rMxW4): ColBERT is a way to make BERT-scale models usable in efficiency-critical applications. The idea is to use two passes through BERT. The first pass involves encoding queries and documents independently with BERT, while the second pass uses a cheap scoring function to perform the final ranking. #RIDER: Reader-Guided Passage Reranking for Open-Domain Question Answering (https://lnkd.in/ex8dWg6K): RIDER introduces a way to boost the performance of open-domain question answering (QA) systems. It uses a reader model to rerank passages that were initially ranked based on the likelihood that they would contain an answer. This model effectively captures dependencies between passages, challenging the conventional view that passages compete rather than cooperate to answer a question. Leveraging LLMs in Scholarly #Knowledge-Graph Question Answering (https://lnkd.in/eTs2m7zn): This paper presents a method for using LLMs to answer complex questions based on scholarly knowledge graphs. The researchers use the LLM to conduct experiments on the scholarly knowledge graph-based OAGKB dataset and prove that the LLM-enriched method effectively boosts question-answering performance. We incorporate many of these techniques at Writer — which is why our accuracy rates are the highest in the LLM industry. We’re hiring software engineers, ML engineers, customer engineers, and solution architects.

RAG Optimization - Episode #1: Current challenges in RAG optimization largely revolve around balancing computational efficiency with the quality of generated content, while also managing the ever-growing size of knowledge bases. The primary need for optimization stems from the necessity to deliver high-quality, contextually relevant responses in real time, without incurring prohibitive computational costs or delays. At Ascendion, we focus on enhancing the speed and accuracy of information retrieval from large datasets, improving the model's ability to integrate this information seamlessly into the generative process, and ensuring the model remains up-to-date with the latest information. Let’s talk about Retrieval Efficiency through Vector Quantization. For retrieval efficiency, a pivotal component is the use of vector quantization techniques within the embedding space to speed up document retrieval. Vector quantization involves mapping high-dimensional vectors (document or query embeddings) into a finite set of vectors in a lower-dimensional space. The objective function for training a vector quantizer can be defined as minimizing the average distortion. Efficient quantization allows for rapid nearest neighbor searches in the quantized space, significantly reducing the computational cost of retrieval from large knowledge bases. Sounds interesting? Reach out to Muthu Chandra to get more insights into RAG optimization. Arun Varadarajan Prakash Balasubramanian Santhosh Mukundan Ramakrishnan JN David Larson Karthik Krishnamurthy Paul Roehrig, PhD Reshma Rahi Pandian Muneeswara C Viral Tripathi Ian Lee (he/him) Joshua Lee #RAG #Optimization #GenerativeAI #cost #quality #schedule

Vector databases are increasingly important in AI, especially for applications using Retrieval-Augmented Generation (RAG). These databases are good at managing and finding complex, high-dimensional data, like the kind used in advanced AI systems. In the context of AI, vector databases are key for embedding-based retrieval (EBR), a process essential for working with language models and unstructured data. This function is crucial for RAG systems, which need to find relevant information and then use it to generate language. This helps AI to give more relevant and precise answers. A recent report, "Survey of Vector Database Management Systems," provides an in-depth analysis of current vector database management systems (VDBMSs). Here's a summary the attached report from researchers from Purdue and Tsinghua Universities 🔍 Introduction to VDBMS: The paper discusses over 20 commercial VDBMSs, focusing on embedding-based retrieval (EBR) and similarity search, driven by large language models and unstructured data needs. 📈 Obstacles in Vector Data Management: Identifies five main challenges: semantic similarity vagueness, vector size, similarity comparison cost, lack of natural partitioning for indexing, and hybrid query difficulties. 🖥️ Techniques in Query Processing: Explores various techniques in query processing, storage, indexing, and optimization, emphasizing the need for low latency, high result quality, and throughput. 📊 Query Interfaces and Optimization: Details query interfaces, optimization, and execution strategies, including hybrid operators and hardware-accelerated query execution. 📚 Review of Current Systems: Classifies current VDBMSs into native systems designed for vectors and extended systems incorporating vector capabilities into existing systems. 📋 Benchmarks and Challenges: Discusses benchmarks for evaluating VDBMSs and outlines several research challenges and directions for future work. 🔮 Conclusion: Concludes with a summary of research challenges and open problems in the field of vector database management systems. It's a good albeit geeky read for those that are interested in how to store and use data alongside large language models.