Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models

By Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, J. Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini and 41 more. arXiv.org, 2024

410 citations recorded by the source corpus.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3462eb6ba2b358ed95afca845e57f22cd0d79a8e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “Part of results are collected from the OpenCompass leaderboard [20] and other papers [2, 30, 32, 148, 176].”
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “Multimodal large language models (MLLMs) [32, 66, 121, 21, 19, 123, 68, 114, 97, 136, 71, 31, 85, 117, 18, 89, 105, 69] have recently achieved or even surpassed human-level performance in a broad spectrum of tasks, underscoring their potential as a significant stride toward artificial general…”
Qwen3-VL Technical Report (2025, positive): “It integrates three sources: (i) public pointing and counting annotations from PixMo (Deitke et al., 2024); (ii) object grounding data derived from public object detection and instance segmentation benchmarks; and (iii) high-precision pointing annotations generated by a dedicated synthesis pipeline…”
π0.5: a Vision-Language-Action Model with Open-World Generalization (2025, positive): “Finally we include a diverse set of web data involving image captioning (CapsFusion [87], COCO [12]), question answering (Cambrian-7M [77], PixMo [19], VQAv2 [32]), and object localization in pre-training.”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, positive): “In recent years, multimodal large language models (MLLMs) [60, 137, 246, 36, 35, 248, 140, 228, 192, 275, 143, 54, 170] have emerged as a pivotal technology in artificial intelligence, capable of processing and understanding information from multiple modalities such as text, images, and videos.”
VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding (2025, positive): “General LLaVA-SFT-665K [38], LLaVA-OV-SI [28], Cambrian-cleaned [39], Pixmo (docs, cap, points, cap-qa, ask-model-anything) [35] 9.87M Document DocVQA [40], Docmatix [41] 1.31M Chart/Figure ChartQA [42], MMC_Instruction [83], DVQA [84], LRV_Instruction [85], ChartGemma [86], InfoVQA [87], PlotQA…”
Leveraging Vision-Language Models for Visual Grounding and Analysis of Automotive UI (2025, unknown): “To establish a suitable baseline model for our fine-tuning and experiments, we evaluated four existing VLMs: TinyClick (0.27B) [24], Molmo-7B-D-0924 [4], UGround-V1-7B (Qwen2-VL-based) [10], and InternVL2 5-8B [2].”