Multimodal Language Generation Techniques

Apple announces MM1 Methods, Analysis & Insights from Multimodal LLM Pre-training In this work, we discuss building performant Multimodal Large Language Models (MLLMs). In particular, we study the importance of various architecture components and data choices. Through careful and comprehensive ablations of the image encoder, the vision language connector, and various pre-training data choices, we identified several crucial design lessons. For example, we demonstrate that for large-scale multimodal pre-training using a careful mix of image-caption, interleaved image-text, and text-only data is crucial for achieving state-of-the-art (SOTA) few-shot results across multiple benchmarks, compared to other published pre-training results. Further, we show that the image encoder together with image resolution and the image token count has substantial impact, while the vision-language connector design is of comparatively negligible importance. By scaling up the presented recipe, we build MM1, a family of multimodal models up to 30B parameters, consisting of both dense models and mixture-of-experts (MoE) variants, that are SOTA in pre-training metrics and achieve competitive performance after supervised fine-tuning on a range of established multimodal benchmarks. Thanks to large-scale pre-training, MM1 enjoys appealing properties such as enhanced in-context learning, and multi-image reasoning, enabling few-shot chain-of-thought prompting.

Curious what might power the intelligence of Apple Vision Pro in the future? 👓 My ex-colleagues from Apple just dropped an exciting new paper on Multimodal LLMs they called MM1. The largest MM1 model (30B dense) achieves state-of-the-art few-shot learning on multimodal benchmarks. 🔍 Key Takeaways: 🔹 𝗔𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗮𝗹 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀: The study emphasizes that the choice of image encoder, particularly image resolution and token count, significantly influences model performance, overshadowing the design of the vision-language connector. 🔹 𝗗𝗮𝘁𝗮 𝗗𝗶𝘃𝗲𝗿𝘀𝗶𝘁𝘆: Incorporating a blend of image-caption, interleaved image-text, and text-only data is critical for state-of-the-art few-shot results. Interestingly, interleaved and text-only data boosts few-shot and text-only performance, while caption data enhances zero-shot capabilities. 🔹 𝗦𝗰𝗮𝗹𝗶𝗻𝗴 𝗦𝘂𝗰𝗰𝗲𝘀𝘀: By strategically scaling model parameters and employing mixture-of-experts (MoE) variants, the MM1 models exhibit competitive performance across multiple multimodal benchmarks after supervised fine-tuning. 🚀 Final Model Recipe: 🔸 𝗢𝗽𝘁𝗶𝗺𝗶𝘇𝗲𝗱 𝗜𝗺𝗮𝗴𝗲 𝗘𝗻𝗰𝗼𝗱𝗲𝗿: Utilizing a ViT-H model with 378x378px resolution pre-trained with a CLIP objective. 🔸 𝗘𝗳𝗳𝗲𝗰𝘁𝗶𝘃𝗲 𝗩𝗶𝘀𝗶𝗼𝗻-𝗟𝗮𝗻𝗴𝘂𝗮𝗴𝗲 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗼𝗿: Leveraging 144 tokens, underscoring the quantity over the architectural design. 🔸 𝗕𝗮𝗹𝗮𝗻𝗰𝗲𝗱 𝗗𝗮𝘁𝗮 𝗠𝗶𝘅: A calculated mixture of 45% interleaved image-text documents, 45% image-text pair documents, and 10% text-only documents ensures robust zero and few-shot performance. The core insight is that deliberate data and architecture choices, not just scale, are key to building performant multimodal models. The MM1 models also exhibit impressive emergent abilities like multi-image reasoning and in-context few-shot learning. Check-out the link in comments below 👇🏼 #AI #MachineLearning #LLM #3MinPapers

Apple recently dropped the MM1 technical report on Multimodal LLM, which is worth over 💲1M. Don't sweat the 41-page length; I've distilled it into a short🔥2-minute summary. Bookmark this for a quick reference: ● What is MM1 about? Diving deep into the optimal training mix for Multimodal LLM, MM1 tackles the blend of architecture (vision encoder and image-text connector), data mix, and training hyperparameters. ● Core Evaluation Tasks: Focusing on image captioning and Visual Question Answering (VQA). ● What training phases does MM1 explore? Visual encoder pre-training -> Multimodal LLM pre-training -> task-specific supervised fine-tuning. ● Insights from Visual Encoder Pre-training: ✅ Image resolution matters most (224->336 with a 3% boost on 0-shot) ✅ Params of the model matter second (ViT 1B -> 3B with 1% boost on 0-shot) ✅ Training data composition (adding a 300M VeCap data brings 1% boost on few-shots) ❌ Contrastive loss or reconstruction in pre-training doesn't notably impact results. ● Vision-Language Model Connector Takeaways: The design complexity of merging vision and text features doesn't matter. The final image token number is the only thing that matters. ● Multimodal LLM Pre-training Data Insights: ❗Integrating text-only and interleaved image-text data enhances few-shot learning capabilities. ❗Optimal data mix for multimodal and text-only tasks is a 5:5:1 ratio of captions, interleaved data, and text. ❗Synthetic caption data bolsters few-shot performance. ● Applicability of Pre-training Insights to Supervised Finetuning: Affirmative. If you want to dive into more details yourself, check this original paper: https://lnkd.in/gzn5ivGy

Text is NOT the only data type we use in RAG pipelines! We are still in the infancy of Generative AI, and text is now the primary information that we feed to LLMs, but this is changing quickly! There is a lot more information contained in the different documents we use on a daily basis beyond just text data. For example, GPT-4, Gemini, and Claude are multimodal LLMs that can ingest images as well as text. The images are passed through a Vision Transformer, resulting in visual tokens. The visual tokens are then passed through a projection layer that specializes in aligning visual tokens with text tokens. The visual and text tokens are then provided to the LLM, which cannot make the difference between the different data modes. In the context of RAG, an LLM plays a role at indexing time, where it generates a vector representation of the data to index it in a vector database. It is also used at retrieval time, where it uses the retrieved documents to provide an answer to a user question. A multimodal LLM can generate embedding representations of images and text and answer questions using those same data types. If we want to answer questions that involve information in different data modes, using a multimodal LLM at indexing and retrieval time is the best option. If you want to build your RAG pipeline using API providers like OpenAI, you can use GPT-4 for question-answering using multimodal prompts. Even if it is available for text generation, it might not be available for embedding generation. Remains creating embedding for images, then? This can be achieved by prompting a multimodal LLM to describe in text the images we need to index. We can then index the images using the text descriptions and their vector representations. The complexity of generating a text description of an image is not the same as answering questions using a large context of different data types. With a small multimodal LLM, we might get satisfactory results in describing images but subpar results in answering questions. For example, it is pretty simple to build an image description pipeline with LlaVA models and Llama.cpp as LLM backbone. Those descriptions can be used for indexing as well as for answering questions that may involve those images. The LLM answering questions would use the text description of images instead of the images themselves. Today, that might be the simplest option for building a multimodal RAG pipeline. It might not be as performant, but the technology is improving very fast! -- 👉 Don't forget to subscribe to my ML newsletter https://lnkd.in/g4iKyRmS --

A new paper introduces Uni-MoE, a large multimodal language model that utilizes a Mixture of Experts (#MoE) architecture to process multiple data modalities like images, speech, video, and text efficiently. Key aspects include: - Modality-specific encoders and connectors map different input modalities into a unified language representation space. - A sparse MoE layer activates only a subset of expert components for each input, enabling efficient scaling. - A three-stage progressive training approach: 1) Cross-modality alignment 2)Training modality-specific experts 3)Tuning the unified multimodal mode Evaluations on multimodal benchmarks for speech recognition, video question-answering, and audio captioning tasks showed Uni-MoE outperforming dense multimodal models like InstructBLIP and Macaw-LLM. The paper demonstrates the potential of using MoE architectures for powerful multimodal AI systems that can understand and process different data modalities efficiently. Learn more about this paper: https://lnkd.in/gFtNSCHg

▶ How amazing is this - Apple's very own Multimodal Language Model! 〽 In the battle of open vs closed-source LLMs, Apple published a paper about their multimodal foundational model, MM1,  with astonishing amount of details. The paper starts with the idea of how important the different architecture and data choices can be in designing today’s LLMs. The authors state that knowledge of the process and principles these LLMs follow are crucial to infer the algorithmic design choices made by the models. And how true is that! Hasn’t the community been asking for it for quite some time? ☘ The main contributions of the paper are: 🔸 Understanding how model architecture and pre-training data affect the performance of the models. Image resolution, visual encoder loss and capacity, and visual encoder pre-training data - in that order - are important while the models are built. 🔸 Three different types of pre-training data are used : image-caption, interleaved image-text, and text-only data. Few-shot performance does well with  interleaved and text-only training data whereas zero performance needs caption data. And this trend holds true during pre-training as well after fine-tuning. Quite interesting, isn’t it? Best thing is after putting these observations together and scaling up the model up to 30B parameters, the model achieves competitive performance across 12 established multimodal benchmarks after Supervised Fine-Tuning (SFT)!  Key Takeaways: 🔺 MM1 uses both dense models (scaling up to 30B parameters) as well as mixture-of-expert (MoE) variants. 🔺 All of image-caption, interleaved image-text, and text-only datasets are important but have different impacts based on performance requirements. 🔺 Image resolution plays a crucial role. 🔺 Pre-training trends hold true after fine-tuning. 💠 What really stood out to me is the staggering amount of details this paper contains page 26 onwards - starting from the actual dataset used followed by the details on the training. We finally have an exuberant and detailed recipe to build a multimodal large language model. With details on three major axes : architecture, data and training process, the MM1 paper is a gold mine for further research and analysis.  It does remain to see though how stable these results hold as the models are scaled further and more variations are incorporated. 📒 P.S. What is your take on open-sourcing these pivotal models? ♻ Please share or repost if you liked reading about MM1! #artificialintelligence #ai #apple #opensource