Multimodal Chain-of-Thought Reasoning in Language Models

By Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, Alexander J. Smola. 2023

914 citations recorded by the source corpus.

Large language models (LLMs) have shown impressive performance on complex reasoning by leveraging chain-of-thought (CoT) prompting to generate intermediate reasoning chains as the rationale to infer the answer. However, existing CoT studies have primarily focused on the language modality. We propose Multimodal-CoT that incorporates language (text) and vision (images) modalities into a two-stage framework that separates rationale generation and answer inference. In this way, answer inference can leverage better generated rationales that are based on multimodal information. Experimental results on ScienceQA and A-OKVQA benchmark datasets show the effectiveness of our proposed approach. With Multimodal-CoT, our model under 1 billion parameters achieves state-of-the-art performance on the ScienceQA benchmark. Our analysis indicates that Multimodal-CoT offers the advantages of mitigating hallucination and enhancing convergence speed. Code is publicly available at https://github.com/amazon-science/mm-cot.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Interestingly, other modalities such as images can be incorporated to improve reasoning capabilities of moderate size LMs (1B) (Zhang et al., 2023), and enabling multimodal chain of thought reasoning (Lu et al., 2022a).”
Visual Instruction Tuning (2023, methodology): “5 model (text-davinci-002) with and without chain-ofthoughts (CoT), LLaMA-Adapter [55], as well as multimodal chain-of-thoughts (MM-CoT) [57], which is the current SoTA method on this dataset.”
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “In Table 2, we compare LLaMA-Adapter with existing popular VQA meth-ods [65, 33, 34] and large language models [27, 4, 74] on ScienceQA datset.”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, methodology): “Existing study [57] have divided such a technique into two categories: Few-ShotCoT [56] and Zero-Shot-CoT [20].”
A survey on multimodal large language models (2023, positive): “Inspired by the success in NLP, multiple works [81,82, 85,86] have been proposed to extend the unimodal CoT to Multimodal CoT (M-CoT).”
Video-R1: Reinforcing Video Reasoning in MLLMs (2025, neutral): “The reasoning abilities of Large Language Models (LLMs) have been a focal point of recent research, aiming to enhance their capacity to perform complex, multi-step problem-solving tasks [22, 31, 27].”
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning (2024, background): “Some recent works, such as multimodal-CoT [76] and [68, 67], have shown improvements by incorporating text-level chain-of-thought reasoning or in-context learning.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “CoT has also been applied to other areas like vision language question answering [257] and code generation [258].”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, neutral): “For instance, multimodal chainof-thought [58] trained a model to incorporate visual inputs as rationales for question answering.”
A Survey of Large Language Models (2023, neutral): “technique to solve multimodal tasks [260] and multilingual tasks [261].”
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic (2023, background): “Mini-GPT4 (Zhu et al., 2023), mPLUG-OWL (Ye et al., 2023), VPGTrans (Zhang et al., 2023a), and InstructBLIP (Dai et al., 2023) retain Q-Former, replace language model to a larger one, and then tuning on meticulously collected instruction data.”
Reasoning with Language Model Prompting: A Survey (2022, neutral): “CoT (Wei et al., 2022b), Self-C (Wang et al., 2022e), STaR (Zelikman et al., 2022), Least-to-Most (Zhang et al., 2022), ZeroCoT (Kojima et al., 2022), Decomposed (Khot et al., 2023), Auto-CoT (Zhang et al., 2023b), PAL (Gao et al., 2022), Teaching (Magister et al., 2022), Fine-tune-CoT (Ho et al., 2”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, background): “Researchers are pioneering the development of LLMs that incorporate text, vision, and other modalities [185].”
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models (2024, methodology): “Inspired by textual chain-of-thought reasoning in LMs [50, 62], S KETCHPAD prompts the underlying visual LM to produce visual artifacts as part of a chain of mixed textual, programmatic, and visual reasoning.”
Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “For instance, multimodal chain of thought prompting involves using diverse sensory and cognitive cues to stimulate thinking and creativity in LLMs [157].”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, methodology): “…instruction or prompting methods , such as Chain-of-Thought (CoT) [123], zero-shot-CoT [61], and ReAct [126], have recently been developed to enhance the performance of LLMs on reasoning, arithmetic and decision making tasks [134, 118, 52, 73, 31, 103, 43, 64, 132, 46, 133, 105, 128, 25, 81, 109].”