Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models

By Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, Dorsa Sadigh. International Conference on Machine Learning, 2024

367 citations recorded by the source corpus.

Visually-conditioned language models (VLMs) have seen growing adoption in applications such as visual dialogue, scene understanding, and robotic task planning; adoption that has fueled a wealth of new models such as LLaVa, InstructBLIP, and PaLI-3. Despite the volume of new releases, key design decisions around image preprocessing, architecture, and optimization are under-explored, making it challenging to understand what factors account for model performance $-$ a challenge further complicated by the lack of objective, consistent evaluations. To address these gaps, we first compile a suite of standardized evaluations spanning visual question answering, object localization, and challenge sets that probe properties such as hallucination; evaluations that provide fine-grained insight VLM capabilities. Second, we rigorously investigate VLMs along key design axes, including pretrained visual representations and training from base vs. instruct-tuned language models, amongst others. We couple our analysis with three resource contributions: (1) a unified framework for evaluating VLMs, (2) optimized, flexible training code, and (3) checkpoints for all models, including a family of VLMs at the 7-13B scale that strictly outperform InstructBLIP and LLaVa v1.5, the state-of-the-art in open VLMs.

View Paper: https://www.semanticscholar.org/paper/f86f71cfd2e9682a56d7334736a7b8a0b1c70b45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

LLaVA-OneVision: Easy Visual Task Transfer (2024, unknown): “For comprehensive investigations on design choices of LMMs, we refer to several recent studies [51, 63, 64, 104, 133, 10].”
Robotic Control via Embodied Chain-of-Thought Reasoning (2024, unknown): “For a given image-instruction pair, we first prompt a Prismatic-7B VLM [35] to describe the scene.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “These include a hierarchical decomposition of robot learning, a multimodal transformer, and discrete latent plans.”
UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, unknown): “To align with Kim et al. [39], our generalist policy is built upon the Prismatic-7B [37] vision-language model (VLM).”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “We use OpenVLA [22] as our representative base VLA, a 7B-parameter manipulation policy created by fine-tuning the Prismatic VLM [20] on 1M episodes from the Open X-Embodiment dataset [33].”
SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model (2025, unknown): “OpenVLA [30] adopts a similar action discretization approach and fine-tune Prismatic VLM [28] only on the OXE dataset [13], which consists of robot data from 22 different robot embodiments across 21 institutions.”
What matters when building vision-language models? (2024, unknown): “It is interesting to note that this finding contradicts (Karamcheti et al., 2024) in which the authors observed that unfreezing the pre-trained visual backbone would significantly degrade the performance.”
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, background): “Various studies [22, 56, 74, 91] have drawn different conclusions regarding the optimal training methodology for MLLMs.”
CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation (2024, unknown): “The success of Large Language Models (LLMs) [2, 9, 63, 64] and Vision-Language Models (VLMs) [1, 14, 28, 37, 61] has inspired the development of Vision-Language-Action (VLA) models, which extend the capabilities of VLMs by integrating action generation.”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, methodology): “We swept the learning rate across multiple orders of magnitude for VLA training, and achieved the best results using a fixed learning rate of 2e-5 (the same learning rate used during VLM pretraining [44]).”
Argus-Retriever: Vision-LLM Late-Interaction Retrieval with Region-Aware Query-Conditioned MoE for Visual Document Retrieval (2026, unknown): “…structural and positional information peaks in middle-to-late transformer layers, before the final layers specialize for the task head (Tenney et al., 2019; Karamcheti et al., 2024), and with the standard VLM practice of taking vision features from a layer before the final one (Liu et al., 2023).”