MultiModalQA: Complex Question Answering over Text, Tables and Images

By Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Hannaneh Hajishirzi, Jonathan Berant. 2021

252 citations recorded by the source corpus.

When answering complex questions, people can seamlessly combine information from visual, textual and tabular sources. While interest in models that reason over multiple pieces of evidence has surged in recent years, there has been relatively little work on question answering models that reason across multiple modalities. In this paper, we present MultiModalQA(MMQA): a challenging question answering dataset that requires joint reasoning over text, tables and images. We create MMQA using a new framework for generating complex multi-modal questions at scale, harvesting tables from Wikipedia, and attaching images and text paragraphs using entities that appear in each table. We then define a formal language that allows us to take questions that can be answered from a single modality, and combine them to generate cross-modal questions. Last, crowdsourcing workers take these automatically-generated questions and rephrase them into more fluent language. We create 29,918 questions through this procedure, and empirically demonstrate the necessity of a multi-modal multi-hop approach to solve our task: our multi-hop model, ImplicitDecomp, achieves an average F1of 51.7 over cross-modal questions, substantially outperforming a strong baseline that achieves 38.2 F1, but still lags significantly behind human performance, which is at 90.1 F1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Question Answering & Reading Comprehension

Citations in the corpus

CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, neutral): “Our work joins a line of previously introduced benchmarks targeting commonsense reasoning 365 [12, 13, 29, 15] as well as probing for other reasoning abilities [32, 33, 34, 35, 36, 37, 38].”
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, background): “…CompWebQ (Talmor and Berant, 2018) Knowledge Graph Question Answer HybridQA (Chen et al., 2020c) Table + Text Passage Question Answer MultiModalQA (Talmor et al., 2021) Table + Text + Image Question Answer FeTaQA (Nan et al., 2021a) et al. (2020) and proposed to convert formal language into an…”
MATE: Multi-view Attention for Table Transformer Efficiency (2021, background): “Recently, more datasets (Kardas et al., 2020; Talmor et al., 2021) requiring parsing larger semi-structured documents have been released.”
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text (2022, methodology): “To study the more realistic open multimodal QA task, WebQA (Chang et al., 2022) and Multi-modalQA (Talmor et al., 2021) MuRAG is built on top of a simpler model we call a “backbone” model, which is pre-trained to encode image-text pairs such that they are suitable for both answer generation and…”
Binding Language Models in Symbolic Languages (2022, background): “We also demonstrate B INDER can be applied on multi-modal knowledge sources (text, table, images, and combined) with M UL - TI M ODAL QA dataset (Talmor et al., 2021).”
FeTaQA: Free-form Table Question Answering (2021, background): “MultiModalQA (Talmor et al., 2021) contains complex questions over text, tables and images.”
Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning (2022, background): “Recent work aims at introducing datasets that require multi-hop reasoning between the textual and tabular data: Hy-bridQA (Chen et al., 2020b), OTTQA (Chen et al., 2020a), MultiModalQA (Talmor et al., 2020), AIT-QA (Katsis et al., 2021), and FeTaQA (Nan et al., 2022).”
♫ MuSiQue: Multihop Questions via Single-hop Question Composition (2021, background): “…challenges such as narrative understanding (Khashabi et al., 2018), discrete reasoning (Dua et al., 2019), multiple modalities (Chen et al., 2020; Talmor et al., 2021), open-domain QA (Geva et al., 2021; Khot et al., 2020; Yang et al., 2018; Talmor and Berant, 2018; Mihaylov et al., 2018) and…”
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding (2021, unknown): “We use multi-hop questions to enforce the constraint of using information from different modalities, which also follows recent work on multimedia QA (Talmor et al. 2021), that however does not require any cross-media grounding.”
Multimodal Learning With Transformers: A Survey (2022, background): “MULTIMODALQA (MMQA) [131], VALUE [132], Fashion IQ [133], LRS2-BBC [134], ActivityNet [135], VisDial [136].”
Large Language Models are few(1)-shot Table Reasoners (2022, background): “…tasks have been proposed like table question answering (Pasu-pat and Liang, 2015; Chen et al., 2020c; Zhu et al., 2021; Chen et al., 2021b; Talmor et al., 2020; Chen et al., 2020a; Nan et al., 2022), table fact veriﬁ-cation (Chen et al., 2019; Aly et al., 2021), table-based generation…”
Invoice Haystack: Benchmarking Document Retrieval and Visual Question Answering Under Strong Visual Homogeneity (2026, unknown): “Early multi-document benchmarks including MultimodalQA [63] and WebQA [11] limited contexts to 10-30 images per question [2, 56], insufficient for enterprise-scale evaluation.”
CFVBench: A Comprehensive Video Benchmark for Fine-grained Multimodal Retrieval-Augmented Generation (2025, unknown): “…existing video-based MRAG methods struggle with detailed multimodal comprehension, which motivates the Adaptive Visual Refinement (AVR) framework that evaluates retrieved visual information and selectively enhances frame sampling or invokes specialized tools to improve fine-grained comprehension.”
Document Haystacks: Vision-Language Reasoning Over Piles of 1000+ Documents (2024, unknown): “The existing multi-image retrieving and reasoning benchmarks are primarily constructed on a small scale, as highlighted in works such as [32, 37].”