Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
By Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, Ashwin Kalyan. 2022
2,503 citations recorded by the source corpus.
When answering a question, humans utilize the information available across different modalities to synthesize a consistent and complete chain of thought (CoT). This process is normally a black box in the case of deep learning models like large-scale language models. Recently, science question benchmarks have been used to diagnose the multi-hop reasoning ability and interpretability of an AI system. However, existing datasets fail to provide annotations for the answers, or are restricted to the textual-only modality, small scales, and limited domain diversity. To this end, we present Science Question Answering (ScienceQA), a new benchmark that consists of ~21k multimodal multiple choice questions with a diverse set of science topics and annotations of their answers with corresponding lectures and explanations. We further design language models to learn to generate lectures and explanations as the chain of thought (CoT) to mimic the multi-hop reasoning process when answering ScienceQA questions. ScienceQA demonstrates the utility of CoT in language models, as CoT improves the question answering performance by 1.20% in few-shot GPT-3 and 3.99% in fine-tuned UnifiedQA. We also explore the upper bound for models to leverage explanations by feeding those in the input; we observe that it improves the few-shot performance of GPT-3 by 18.96%. Our analysis further shows that language models, similar to humans, benefit from explanations to learn from fewer data and achieve the same performance with just 40% of the data. The data and code are available at https://scienceqa.github.io.
Subfields: Artificial Intelligence, Prompting & Reasoning, Multimodal Learning, Question Answering & Reading Comprehension
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Caiming Xiong
- Yoshua Bengio
- Quoc V. Le
- Noah A. Smith
In Prompting & Reasoning
In Multimodal Learning
In Question Answering & Reading Comprehension
Citations in the corpus
- Visual Instruction Tuning (2023, methodology): “We study our method on the ScienceQA benchmark [34], the first large-scale multimodal science question dataset that annotates the answers with detailed lectures and explanations.”
- Improved Baselines with Visual Instruction Tuning (2023, methodology): “Following InstructBLIP [13], the image subset of ScienceQA [37] with multiple choice are used to evaluate the zero-shot generalization on scientific question answering.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “We adopt greedy search as the decoding method for generation, and keep other hyperparameters the same as the instruction-following LLaMA-Adapter.”
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, neutral): “MM-CoT [37] finetunes on the reasoning chain annotated in the ScienceQA [21] dataset to achieve the CoT capability in the science question answering task.”
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, positive): “Specifically, we adopt UnifiedQA (Khashabi et al., 2020) to initialize our models in the two stages because it achieves the best fine-tuning results in Lu et al. (2022a).”
- SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, neutral): “As we can see, on most datasets, i.e. , ScienceQA (Lu et al., 2022), TextQA (Singh et al., 2019), RefCOCO (Lin et al., 2014) and Mathvista (Lu et al., 2023), when activating two experts, keeping it the same with training setting, SPHINX-MoE performs the best.”
- To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (2024, background): “ScienceQA (Lu et al., 2022) consists of multiple choice questions across a range of natural and social science disciplines, though it is hard to interpret gains without knowing breaking down performance by subject or question type.”
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “…(en&zh) [18] VQAv2 (en) [28], GQA (en) [34], OKVQA (en) [76], General QA VSR (en) [55], VisualDialog (en) [24] Science AI2D (en) [39], ScienceQA (en) [69], TQA (en) [40] ChartQA (en) [77], MMC-Inst (en) [57], DVQA (en) [38], Chart PlotQA (en) [81], LRV-Instruction (en) [56] GeoQA+ (en) [14],…”
- SVIT: Scaling up Visual Instruction Tuning (2023, background): “…evaluate these models on popular benchmarks: VQA-v2 (Goyal et al., 2017), GQA (Hudson & Manning, 2019), VisWiz (Gurari et al., 2018), ScienceQA-IMG (Lu et al., 2022), TextVQA (Singh et al., 2019), MME perception (Fu et al., 2023), MME cognition (Fu et al., 2023), MMBench (Liu et al., 2023c),…”
- WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models (2024, background): “…by GPT-4V (OpenAI, 2023) and Gemini (Team et al., 2023), LMMs have demonstrated impressive performance on benchmarks (Goyal et al., 2017; Lu et al., 2022; Zellers et al., 2019; Hes-sel et al., 2022), establishing a foundation for the construction of multimodal agents in subsequent…”
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024, background): “…from 165k images by prompting 21], Infograph-icVQA train [22], AI2D train (transparent and opaque label boxes) [13], A-OKVQA train [28], Android-Control train [14], ScienceQA train [16], TabMWP train [17], ST-VQA train [6], TallyQA train [3], DVQA train [11], FigureQA train [12], and train [23].”
- Generative Multimodal Models are In-Context Learners (2023, methodology): “A summary of data used is as follows: ( a ) Academic-task-oriented data: image cap-tioning [18, 66], visual question answering [28, 32, 68], knowledgeable question answering [51, 53], multimodal classification [45], and referring expression comprehen-4 sion [34, 52].”
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (2024, methodology): “In the existing data sources, there are 32k samples in PathVQA [46] and 12k in ScienceQA [84].”
- LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models (2024, positive): “With the advent of Large Multimodal Models (LMMs), these have increasingly focused on broader capabilities spanning both vision and language, including reasoning [48] and visual instruction following [42].”