Multimodal Reasoning Techniques

🚀 Excited to introduce MultiModal Graph Learning (MMGL) - exploring beyond the typical one-to-one data modality pairs to uncover more complex and multifaceted relationships among data modalities! 🌐 Diving into REAL-WORLD settings where entities of different modalities interact in multifaceted ways, we propose a novel approach: Representing these complex relationships as GRAPHS, capable of capturing ANY number of modalities & variable relationships! 🎯 Focused on generative tasks and wielding the power of pretrained Language Models, we navigate how to infuse multiple neighbor info & graph structure info into the LMs without scalability issues! 🔎 MMGL brings 3 principled Research Questions: 1️⃣ How to encode multiple neighbor info into pretrained LMs without scalability issues? 2️⃣ How to encode the graph structure info among multimodal neighbors into LMs? 3️⃣ How to finetune LMs to learn from neighbor context parameter-efficiently? 📊 📑 Our extensive experiment and in-depth analysis find: 1️⃣ Neighbor context enhances the generation. 2️⃣ Text embeddings are the key to scalability issues, yet not as effective as raw texts. 3️⃣ GNN embeddings lead in graph structure encodings. 4️⃣ LoRA & Flamingo top PEFT models' performance. 💫 Our work not only answers pivotal questions raised in MMGL but also lays down a solid foundation for future MMGL research. Hope the multifaceted fusion of different modalities can unlock doors to more comprehensive AI models! 🤖 👉 Paper: https://lnkd.in/dKExqSZ8 👏 Huge thanks to my amazing collaborators Ruslan Salakhutdinov, Bryan Hooi, Jing Yu Koh

Apple recently dropped the MM1 technical report on Multimodal LLM, which is worth over 💲1M. Don't sweat the 41-page length; I've distilled it into a short🔥2-minute summary. Bookmark this for a quick reference: ● What is MM1 about? Diving deep into the optimal training mix for Multimodal LLM, MM1 tackles the blend of architecture (vision encoder and image-text connector), data mix, and training hyperparameters. ● Core Evaluation Tasks: Focusing on image captioning and Visual Question Answering (VQA). ● What training phases does MM1 explore? Visual encoder pre-training -> Multimodal LLM pre-training -> task-specific supervised fine-tuning. ● Insights from Visual Encoder Pre-training: ✅ Image resolution matters most (224->336 with a 3% boost on 0-shot) ✅ Params of the model matter second (ViT 1B -> 3B with 1% boost on 0-shot) ✅ Training data composition (adding a 300M VeCap data brings 1% boost on few-shots) ❌ Contrastive loss or reconstruction in pre-training doesn't notably impact results. ● Vision-Language Model Connector Takeaways: The design complexity of merging vision and text features doesn't matter. The final image token number is the only thing that matters. ● Multimodal LLM Pre-training Data Insights: ❗Integrating text-only and interleaved image-text data enhances few-shot learning capabilities. ❗Optimal data mix for multimodal and text-only tasks is a 5:5:1 ratio of captions, interleaved data, and text. ❗Synthetic caption data bolsters few-shot performance. ● Applicability of Pre-training Insights to Supervised Finetuning: Affirmative. If you want to dive into more details yourself, check this original paper: https://lnkd.in/gzn5ivGy

Curious what might power the intelligence of Apple Vision Pro in the future? 👓 My ex-colleagues from Apple just dropped an exciting new paper on Multimodal LLMs they called MM1. The largest MM1 model (30B dense) achieves state-of-the-art few-shot learning on multimodal benchmarks. 🔍 Key Takeaways: 🔹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: The study emphasizes that the choice of image encoder, particularly image resolution and token count, significantly influences model performance, overshadowing the design of the vision-language connector. 🔹 𝗗𝗮𝘁𝗮 𝗗𝗶𝘃𝗲𝗿𝘀𝗶𝘁𝘆: Incorporating a blend of image-caption, interleaved image-text, and text-only data is critical for state-of-the-art few-shot results. Interestingly, interleaved and text-only data boosts few-shot and text-only performance, while caption data enhances zero-shot capabilities. 🔹 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗦𝘂𝗰𝗰𝗲𝘀𝘀: By strategically scaling model parameters and employing mixture-of-experts (MoE) variants, the MM1 models exhibit competitive performance across multiple multimodal benchmarks after supervised fine-tuning. 🚀 Final Model Recipe: 🔸 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗜𝗺𝗮𝗴𝗲 𝗘𝗻𝗰𝗼𝗱𝗲𝗿: Utilizing a ViT-H model with 378x378px resolution pre-trained with a CLIP objective. 🔸 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗩𝗶𝘀𝗶𝗼𝗻-𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿: Leveraging 144 tokens, underscoring the quantity over the architectural design. 🔸 𝗕𝗮𝗹𝗮𝗻𝗰𝗲𝗱 𝗗𝗮𝘁𝗮 𝗠𝗶𝘅: A calculated mixture of 45% interleaved image-text documents, 45% image-text pair documents, and 10% text-only documents ensures robust zero and few-shot performance. The core insight is that deliberate data and architecture choices, not just scale, are key to building performant multimodal models. The MM1 models also exhibit impressive emergent abilities like multi-image reasoning and in-context few-shot learning. Check-out the link in comments below 👇🏼 #AI #MachineLearning #LLM #3MinPapers

Apple announces MM1 Methods, Analysis & Insights from Multimodal LLM Pre-training In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

A new paper introduces Uni-MoE, a large multimodal language model that utilizes a Mixture of Experts (#MoE) architecture to process multiple data modalities like images, speech, video, and text efficiently. Key aspects include: - Modality-specific encoders and connectors map different input modalities into a unified language representation space. - A sparse MoE layer activates only a subset of expert components for each input, enabling efficient scaling. - A three-stage progressive training approach: 1) Cross-modality alignment 2)Training modality-specific experts 3)Tuning the unified multimodal mode Evaluations on multimodal benchmarks for speech recognition, video question-answering, and audio captioning tasks showed Uni-MoE outperforming dense multimodal models like InstructBLIP and Macaw-LLM. The paper demonstrates the potential of using MoE architectures for powerful multimodal AI systems that can understand and process different data modalities efficiently. Learn more about this paper: https://lnkd.in/gFtNSCHg