AI Techniques For Document Image Recognition

💡 Sharing my view on #OCR vs #LLM. I recently saw quite a few people proclaim the death of OCR with the rise of LLMs, the reality is more nuanced. LLM is a natural progression from technology standpoint; it is better, cheaper, and improving rapidly. However, a lot of people use the term OCR in a very broad context, i.e. translating texts on images or documents into meaningful knowledge and concept. I'm particularly excited to see #DocumentAI, a project I started in 2021 before the LLM boom, being recognized as a key benchmark in the latest Mistral OCR announcement. It is still considered as state-of-the-arts today, because it solves some of the most challenging document types that people typically do not want to benchmark, like payslip, insurance doc, commercial contract, etc. Our vision was to develop a generalized document model on common document types, and we pioneered the "Uptrain" concept for tailoring parsers to specific documents. Two lessons I learnt from this experience: 1. Pass-through rate is all it matters: In document processing, achieving a high pass-through rate is critical. The difference between a system that gets 9 in 10 fields right on one document vs. one that correctly processes 9 out of 10 documents 100% is substantial. Although in both cases, a benchmark will tell you the accuracy is 90%, the latter enables complete automation, and the former requires human to look at every document. 2. Entity resolution transform text into knowledge: we knew that a typically property insurance document contains approximately 6-9 different addresses. Entity resolution disambiguates between mailing and insured addresses, which can be fed into downstream system for automation. Looking at this $7.8B document processing industry, there's a significant market for applying LLM techniques to parsing and extraction, providing substantial value further up the document processing chain. It is not yet a solved problem. So, what does this all mean? 1. LLMs represent a natural and promising evolution from legacy OCR, offering improved accuracy and reduced costs over time. 2. Take a grain-of-salt when looking at OCR benchmark, understanding the quality number might not directly translate into cost saving 3. Bounding box(reporting actual coordinates) is not yet at the same quality of a OCR. #Gemini Flash 2.0 has first-of-its-kind spatial understanding, and can return pixel coordinates. However, it is not perfect, and we still have work to do. 4. LLMs are likely sufficient for consumer apps seeking basic text extraction from images, such as phone numbers or addresses. It is the go-to place for all developers. 5. LLMs alone is not enough to solve the broader document processing and automation problem. While agentic framework helps, achieving high pass-through rates requires a lot more additional work, and also put AI into complex review and escalation pipelines optimized for efficiency.

There's a paper making waves in the AI Retrieval community that addresses some fundamental challenges in multimodal document retrieval. Given the buzz, I thought it was worth unpacking its implications, especially since I've been discussing RAG systems and their limitations. The paper, by Manuel Faysse et al, introduces ColPali, a novel retrieval model, and ViDoRe, a benchmark for visually rich documents. Here's why it's generating so much attention: 1. ColPali processes documents directly from images, sidestepping the complex parsing pipelines we're all too familiar with. 2. It employs a "late interaction" mechanism, matching query components to document image patches. This approach, inspired by ColBERT, allows for more nuanced retrieval. 3. It's built on PaliGemma-3B, a vision-language model that combines SigLIP patch embeddings with a Gemma-2B language model. This architecture enables efficient processing of both textual and visual information. 4. The model uses a projection layer to map embeddings to a 128-dimensional space, balancing performance and efficiency. Performance surpasses all other methods, with faster indexing times. The late interaction operator, similar to ColBERT, allows for fine-grained matching between query tokens and document patches. I've had the chance to experiment with ColPali, and I have to say, it looks very strong. The results are impressive, particularly in handling complex document layouts. What's striking is how this addresses the longstanding issue of handling documents with tables, charts, and complex layouts - a pain point in building comprehensive RAG systems. This could be a game-changer for teams grappling with large, multimodal document collections, especially in domains like healthcare or financial analysis. paper: https://lnkd.in/egW_zUEx HF space: https://lnkd.in/ei2bG3zA Model: https://lnkd.in/eXyQxkEu #AIResearch #DocumentRetrieval #RAG

I finally had the chance to explore a new document extraction technique introduced in a paper last September. Bonus: the code and model are free to use (Apache 2.0). This new approach, called General OCR Theory (GOT-OCR2.0), suggests a unified end-to-end model that handles tasks traditional OCR systems struggle with. Unlike legacy OCR, which relies on complex multi-modular pipelines, GOT uses a simple encoder-decoder architecture with only 580M parameters that outperforms models 10-100× larger. Paper highlights: (1) Unified architecture - a high-compression encoder paired with a long-context decoder that handles everything from scene text to complex formulas (2) Stunning performance - delivers nearly perfect text accuracy on documents, surpassing Qwen-VL-Max (>72B) and other leading models (3) Versatility beyond text - processes math formulas, molecular structures, and even geometric shapes (4) Interactive capabilities - supports region-level recognition guided by coordinates or colors I just tried it out and was blown away by how it handles complex documents with mixed content types. The ability to convert math formulas from Arxiv PDFs to Mathpix format alone is worth exploring this model. What strikes me most about GOT is how it challenges the notion that only billion-parameter LLMs can tackle complex visual tasks. Paper + code + model can be found in their GitHub repo https://lnkd.in/dbHzUUYx — Join thousands of world-class researchers and engineers from Google, Stanford, OpenAI, and Meta staying ahead on AI http://aitidbits.ai