AmbigQA: Answering Ambiguous Open-domain Questions

By Sewon Min, Julian Michael, Hannaneh Hajishirzi, Luke S. Zettlemoyer. 2020

498 citations recorded by the source corpus.

Ambiguity is inherent to open-domain question answering; especially when exploring new topics, it can be difficult to ask questions that have a single, unambiguous answer. In this paper, we introduce AmbigQA, a new open-domain question answering task which involves predicting a set of question-answer pairs, where every plausible answer is paired with a disambiguated rewrite of the original question. To study this task, we construct AmbigNQ, a dataset covering 14,042 questions from NQ-open, an existing open-domain QA benchmark. We find that over half of the questions in NQ-open are ambiguous, exhibiting diverse types of ambiguity. We also present strong baseline models for AmbigQA which we show benefit from weakly supervised learning that incorporates NQ-open, strongly suggesting our new task and data will support significant future research effort. Our data is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “By processing passages independently in the encoder, but jointly in the decoder, this method differs from Min et al. (2020) and Lewis et al. (2020).”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Recently, generating the answer as free-form text, using a seq2seq model conditioned on retrieved documents have become prevalent (Lewis et al., 2020; Izacard & Grave, 2020; Min et al., 2020).”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “(Mitra et al., 2020) RoBERTa (Clark et al.,2019c) FreeLB-RoBERTa (Zhu et al., 2020) RoBERTa (Clark et al.,2019c) FreeLB-RoBERTa (Zhu et al., 2020) -- KF+SIR +2Step (Mitra et al., 2020) 75.7 (Lan et al.,2019) FreeLB-RoBERTa (Zhu et al.,2020) RoBERTa (Sakaguchi et al.,2019) RoBERTa (Bisk et al., 2019)”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, background): “The generative reader generates an answer while conditioned on question alone (Roberts et al., 2020), or question with relevant passages (Lewis et al., 2020; Min et al., 2020).”
A Memory Efficient Baseline for Open Domain Question Answering (2020, methodology): “Several works showed that generative QA models could be improved by retrieval, using the obtained documents as input of the sequenceto-sequence model (Min et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020b).”
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “…has been used for various purposes, such as data augmentation (Alberti et al., 2019; Lewis et al., 2019b; Lee et al., 2021), improved retrieval (Nogueira et al., 2019), generative modelling for contextual QA (Lewis and Fan, 2018), as well as being studied in its own right (Du et al., 2017;…”
QED: A Framework and Dataset for Explanations in Question Answering (2020, background): “It would be natural for a system to provide two possible answers (see, e.g. Min et al., 2020), with two possible QED explanations highlighting the differing assumptions underlying each answer.”
Lost in the Middle: How Language Models Use Long Contexts (2023, methodology): “We use the ambiguity annotations of Min et al. (2020) to create a subset unambiguous questions.”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, background): “change over time (Min et al., 2020), no prior work investigated the performance of existing models on temporally dependent queries and their prevalence.”
Is Retriever Merely an Approximator of Reader? (2020, unknown): “…- - DPR-Single (Karpukhin et al., 2020) 41.5 56.8 - ë w/ RDR 42.1 (+0.6) 57.0 (+0.2) BM25+DPR-Multi (Karpukhin et al., 2020) 38.8 57.9 - SpanSeqGen (Min et al., 2020) 42.5 - - RAG-Token (Lewis et al., 2020) 44.1 55.2 66.1 ë w/ RDR 44.5 (+0.4) - - RAG-Sequence (Lewis et al., 2020) 44.5 56.1 68.0 T5…”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, methodology): “QA-FEEDBACK is based on ASQA [39], a dataset that focuses on answering ambiguous factoid questions [26] in an open-domain setting.”
Task-aware Retrieval with Instructions (2022, background): “Ambig QA (Min et al., 2020) 1,172 18,809 Wikipedia question duplicated question WIKIQA (Yang et al.”
A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models (2024, unknown): “More specifically, to facilitate the generation task in the NLP area, RAG incorporates information or knowledge from external data sources, which serves as supplementary reference/instruction for the input query or the generated output [56, 87].”
ANLIzing the Adversarial Natural Language Inference Dataset (2020, background): “in human judgements for this task (Pavlick and Kwiatkowski, 2019; Min et al., 2020; Nie et al., 2020b).”
Reader-Guided Passage Reranking for Open-Domain Question Answering (2021, background): “…BART-large (Lewis et al., 2019), which concatenates the question and top-10 retrieved passages (without reranking during training), trims them to 1,024 tokens (7.8 passages are left on average) as the input, and learns to generate the answer in a Seq2Seq manner (Mao et al., 2020; Min et al., 2020).”
Enabling Large Language Models to Generate Text with Citations (2023, background): “As shown in Figure 1, each question is an ambiguous question from AmbigQA (Min et al., 2020) that requires multiple short answers to cover different aspects, and the dataset provides a long-form answer that covers all short answers.”
Generate rather than Retrieve: Large Language Models are Strong Context Generators (2022, methodology): “As the temporal effects on the NQ dataset have been observed in many other recent papers (Min et al., 2020; Zhang & Choi, 2021), we conduct ablation studies mainly on the TriviaQA and WebQ datasets, to avoid the bias of the experimental results caused by the temporality issue.”