LlaVA-CoT: Let Vision Language Models Reason Step-By-Step

By Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, Li Yuan. IEEE International Conference on Computer Vision, 2024

528 citations recorded by the source corpus.

Large language models have demonstrated substantial advancements in reasoning capabilities. However, current Vision-Language Models (VLMs) often struggle to perform systematic and structured reasoning, especially when handling complex visual question-answering tasks. In this work, we introduce LLaVA-COT11Our LLaVA-CoT is built upon Llama-3.2-Vision model [43]., a large VLM designed to conduct autonomous multistage reasoning. Unlike chain-of-thought prompting, LLaVA-COT independently engages in sequential stages of summarization, visual interpretation, logical reasoning, and conclusion generation. This structured approach enables LLaVA-CoT to achieve marked improvements on reasoning-intensive tasks. To accomplish this, we construct the LLaVA-COT-100k dataset, integrating samples from various visual question answering sources and providing structured reasoning annotations. Besides, we propose a test-time stage-wise retracing search method (SWIRES), which enables effective and efficient testtime scaling. Remarkably, with only 100 k training samples and test-time scaling, LLaVA-COT not only outperforms its base model by $\mathbf{9. 4 \%}$ on a wide range of multimodal reasoning benchmarks, but also surpasses the performance of larger and even closed-source models, such as Gemini-1.5pro, GPT-4o-mini, and Llama-3.2-90B-Vision-Instruct. The code, dataset, and pre-trained weights are publicly available at https://github.com/PKU-YuanGroup/LLaVA-CoT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b368528a18d8f7b377e6fc74c1050df8c0348a1f

Citations in the corpus

• Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, unknown): “Many existing works [15, 17] have attempted to construct multimodal reasoning datasets to enhance the reasoning capability of MLLMs. Prior efforts typically gathered CoT data which often lack the natural cognitive processes of questioning, and self-reflection.”
• R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, unknown): “For example, LLaVA-COT [41] prompts GPT-4 to generate CoT data with structured reasoning templates, i.e., sum-mary, caption, reasoning and conclusion, and then performs SFT on the generated data.”
• R1-Onevision: Advancing Generalized Multimodal Reasoning Through Cross-Modal Formalization (2025, unknown): “LLaVA-CoT [31] introduces a novel VLM designed to conduct visual reasoning, demonstrating exceptional performance on tasks that require structured thinking and reasoning.”
• Mieb: Massive Image Embedding Benchmark (2025, unknown): “We hypothesize that future models that better incorporate reasoning capabilities and test-time scaling techniques [20, 24, 37, 43, 62] may achieve better results on compositionality tasks.”