ComQA: A Community-sourced Dataset for Complex Factoid Question Answering with Paraphrase Clusters

By Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, Gerhard Weikum. 2018

70 citations recorded by the source corpus.

To bridge the gap between the capabilities of the state-of-the-art in factoid question answering (QA) and what users ask, we need large datasets of real user questions that capture the various question phenomena users are interested in, and the diverse ways in which these questions are formulated. We introduce ComQA, a large dataset of real user questions that exhibit different challenging aspects such as compositionality, temporal reasoning, and comparisons. ComQA questions come from the WikiAnswers community QA platform, which typically contains questions that are not satisfactorily answerable by existing search engine technology. Through a large crowdsourcing effort, we clean the question dataset, group questions into paraphrase clusters, and annotate clusters with their answers. ComQA contains 11,214 questions grouped into 4,834 paraphrase clusters. We detail the process of constructing ComQA, including the measures taken to ensure its high quality while making effective use of crowdsourcing. We also present an extensive analysis of the dataset and the results achieved by state-of-the-art systems on ComQA, demonstrating that our dataset can be a driver of future research on QA.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Retrieval & Search

In Natural Language Processing

Citations in the corpus

Break It Down: A Question Understanding Benchmark (2020, background): “• Reading Comprehension (RC): Questions that require understanding of a text passage by reasoning over multiple sentences (Talmor and Berant, 2018; Yang et al., 2018; Dua et al., 2019; Abujabal et al., 2019).”
Exploring and Predicting Transferability across NLP Tasks (2020, background): “, 2019) 16K ComQA (Abujabal et al., 2019) 11K CQ (Bao et al.”
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension (2019, methodology): “…(Tay et al., 2018) for N EWS QA, in Lin et al. (2018), for S EARCH QA, in Das et al. (2019) for TQA-U, in (Talmor and Berant, 2018b) for CWQ, in Ding et al. (2019) for H OTPOT QA, in (Abujabal et al., 2018) for C OM QA, and in Bao et al. (2016) for CQ. train on all examples in the large datasets.”
Constrained Language Models Yield Few-Shot Semantic Parsers (2021, background): “, 2018), reading comprehension (Talmor and Berant, 2018; Yang et al., 2018; Dua et al., 2019; Abujabal et al., 2019), and visual question answering (Johnson et al.”
MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answering (2020, methodology): “To avoid introducing such biases and to be able to fully utilize our set of 10,000 translated queries, we use an annotation scheme similar to Abujabal et al. (2019), in which the answer is annotated directly.”
Minimizing Human Intervention in Online Classification (2025, unknown): “By design, ComQA emphasizes reasoning-intensive phenomena—namely compositional queries, explicit temporal constraints, and comparative or superlative constructions—making it a rigorous benchmark for research on complex factoid question answering.”