Improving Multimodal Model Performance

Apple announces MM1 Methods, Analysis & Insights from Multimodal LLM Pre-training In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

Apple recently dropped the MM1 technical report on Multimodal LLM, which is worth over 💲1M. Don't sweat the 41-page length; I've distilled it into a short🔥2-minute summary. Bookmark this for a quick reference: ● What is MM1 about? Diving deep into the optimal training mix for Multimodal LLM, MM1 tackles the blend of architecture (vision encoder and image-text connector), data mix, and training hyperparameters. ● Core Evaluation Tasks: Focusing on image captioning and Visual Question Answering (VQA). ● What training phases does MM1 explore? Visual encoder pre-training -> Multimodal LLM pre-training -> task-specific supervised fine-tuning. ● Insights from Visual Encoder Pre-training: ✅ Image resolution matters most (224->336 with a 3% boost on 0-shot) ✅ Params of the model matter second (ViT 1B -> 3B with 1% boost on 0-shot) ✅ Training data composition (adding a 300M VeCap data brings 1% boost on few-shots) ❌ Contrastive loss or reconstruction in pre-training doesn't notably impact results. ● Vision-Language Model Connector Takeaways: The design complexity of merging vision and text features doesn't matter. The final image token number is the only thing that matters. ● Multimodal LLM Pre-training Data Insights: ❗Integrating text-only and interleaved image-text data enhances few-shot learning capabilities. ❗Optimal data mix for multimodal and text-only tasks is a 5:5:1 ratio of captions, interleaved data, and text. ❗Synthetic caption data bolsters few-shot performance. ● Applicability of Pre-training Insights to Supervised Finetuning: Affirmative. If you want to dive into more details yourself, check this original paper: https://lnkd.in/gzn5ivGy

Curious what might power the intelligence of Apple Vision Pro in the future? 👓 My ex-colleagues from Apple just dropped an exciting new paper on Multimodal LLMs they called MM1. The largest MM1 model (30B dense) achieves state-of-the-art few-shot learning on multimodal benchmarks. 🔍 Key Takeaways: 🔹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: The study emphasizes that the choice of image encoder, particularly image resolution and token count, significantly influences model performance, overshadowing the design of the vision-language connector. 🔹 𝗗𝗮𝘁𝗮 𝗗𝗶𝘃𝗲𝗿𝘀𝗶𝘁𝘆: Incorporating a blend of image-caption, interleaved image-text, and text-only data is critical for state-of-the-art few-shot results. Interestingly, interleaved and text-only data boosts few-shot and text-only performance, while caption data enhances zero-shot capabilities. 🔹 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗦𝘂𝗰𝗰𝗲𝘀𝘀: By strategically scaling model parameters and employing mixture-of-experts (MoE) variants, the MM1 models exhibit competitive performance across multiple multimodal benchmarks after supervised fine-tuning. 🚀 Final Model Recipe: 🔸 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗜𝗺𝗮𝗴𝗲 𝗘𝗻𝗰𝗼𝗱𝗲𝗿: Utilizing a ViT-H model with 378x378px resolution pre-trained with a CLIP objective. 🔸 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗩𝗶𝘀𝗶𝗼𝗻-𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿: Leveraging 144 tokens, underscoring the quantity over the architectural design. 🔸 𝗕𝗮𝗹𝗮𝗻𝗰𝗲𝗱 𝗗𝗮𝘁𝗮 𝗠𝗶𝘅: A calculated mixture of 45% interleaved image-text documents, 45% image-text pair documents, and 10% text-only documents ensures robust zero and few-shot performance. The core insight is that deliberate data and architecture choices, not just scale, are key to building performant multimodal models. The MM1 models also exhibit impressive emergent abilities like multi-image reasoning and in-context few-shot learning. Check-out the link in comments below 👇🏼 #AI #MachineLearning #LLM #3MinPapers

▶ How amazing is this - Apple's very own Multimodal Language Model! 〽 In the battle of open vs closed-source LLMs, Apple published a paper about their multimodal foundational model, MM1,  with astonishing amount of details. The paper starts with the idea of how important the different architecture and data choices can be in designing today’s LLMs. The authors state that knowledge of the process and principles these LLMs follow are crucial to infer the algorithmic design choices made by the models. And how true is that! Hasn’t the community been asking for it for quite some time? ☘ The main contributions of the paper are: 🔸 Understanding how model architecture and pre-training data affect the performance of the models. Image resolution, visual encoder loss and capacity, and visual encoder pre-training data - in that order - are important while the models are built. 🔸 Three different types of pre-training data are used : image-caption, interleaved image-text, and text-only data. Few-shot performance does well with  interleaved and text-only training data whereas zero performance needs caption data. And this trend holds true during pre-training as well after fine-tuning. Quite interesting, isn’t it? Best thing is after putting these observations together and scaling up the model up to 30B parameters, the model achieves competitive performance across 12 established multimodal benchmarks after Supervised Fine-Tuning (SFT)!  Key Takeaways: 🔺 MM1 uses both dense models (scaling up to 30B parameters) as well as mixture-of-expert (MoE) variants. 🔺 All of image-caption, interleaved image-text, and text-only datasets are important but have different impacts based on performance requirements. 🔺 Image resolution plays a crucial role. 🔺 Pre-training trends hold true after fine-tuning. 💠 What really stood out to me is the staggering amount of details this paper contains page 26 onwards - starting from the actual dataset used followed by the details on the training. We finally have an exuberant and detailed recipe to build a multimodal large language model. With details on three major axes : architecture, data and training process, the MM1 paper is a gold mine for further research and analysis.  It does remain to see though how stable these results hold as the models are scaled further and more variations are incorporated. 📒 P.S. What is your take on open-sourcing these pivotal models? ♻ Please share or repost if you liked reading about MM1! #artificialintelligence #ai #apple #opensource