Hurdles to Progress in Long-form Question Answering

By Kalpesh Krishna, Aurko Roy, Mohit Iyyer. 2021

237 citations recorded by the source corpus.

The task of long-form question answering (LFQA) involves retrieving documents relevant to a given question and using them to generate a paragraph-length answer. While many models have recently been proposed for LFQA, we show in this paper that the task formulation raises fundamental challenges regarding evaluation and dataset creation that currently preclude meaningful modeling progress. To demonstrate these challenges, we first design a new system that relies on sparse attention and contrastive retriever learning to achieve state-of-the-art performance on the ELI5 LFQA dataset. While our system tops the public leaderboard, a detailed analysis reveals several troubling trends: (1) our system’s generated answers are not actually grounded in the documents that it retrieves; (2) ELI5 contains significant train / validation overlap, as at least 81% of ELI5 validation questions occur in paraphrased form in the training set; (3) ROUGE-L is not an informative metric of generated answer quality and can be easily gamed; and (4) human evaluations used for other text generation tasks are unreliable for LFQA. We offer suggestions to mitigate each of these issues, which we hope will lead to more rigorous LFQA research and meaningful progress in the future.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “…Zellers et al., 2019), summarization (Gabriel et al., 2021; Maynez et al., 2020; Stiennon et al., 2020; Wang et al., 2020), conversational dialog (Shuster et al., 2021; Roller et al., 2021), and question answering (Dou et al., 2021; Krishna et al., 2021; Lewis et al., 2020; Logan IV et al., 2019).”
Augmented Language Models: a Survey (2023, background): “However, although this conclusion is straightforward when equipping a LM with a calculator, there is surprisingly little evidence of it for information retrieval augmented LMs (Krishna et al., 2021).”
WebGPT: Browser-assisted question-answering with human feedback (2021, background): “LFQA systems have the potential to become one of the main ways people learn about the world, but currently lag behind human performance [Krishna et al., 2021].”
Survey of Hallucination in Natural Language Generation (2022, methodology): “Normally GQA system involves searching an external knowledge source for information relevant to the question.”
Entity-Based Knowledge Conflicts in Question Answering (2021, background): “Overreliance on Parametric Knowledge Krishna et al. (2021) showed that replacing the retrieved documents with random documents during inference yields similar performance for the task of long form question answering.”
RankGen: Improving Text Generation with Large Ranking Models (2022, background): “, 2022) alleviate these issues, they can also produce text with inconsistencies, hallucinations, factual errors, or commonsense issues (Massarelli et al., 2020; Dou et al., 2022; Krishna et al., 2021).”
Attributed Text Generation via Post-hoc Research and Revision (2022, background): “For example, prior work has shown that RAMs generate text that either includes additional information not present in the retrieved document (Dziri et al., 2022), ignores the document altogether (Krishna et al., 2021), or even contradicts the document (Longpre et al., 2021).”
FRUIT: Faithfully Reflecting Updated Information in Text (2021, background): “Whenever the provided evidence contradicts parametric knowledge, the model must prefer the evidence, which recent work has shown is difficult for pretrained language models (Krishna et al., 2021; Longpre et al., 2021).”
RARR: Researching and Revising What Language Models Say, Using Language Models (2022, background): “, 2022), ignores the document altogether (Krishna et al., 2021), or even contradicts the document (Longpre et al.”
A Survey of Knowledge-enhanced Text Generation (2020, methodology): “For training the retriever and generator, most of existing work has jointly optimized these two components without any direct supervision on what document should be retrieve [61, 64].”
Enabling Large Language Models to Generate Text with Citations (2023, methodology): “Evaluating the correctness of long-form generation is a challenging task (Krishna et al., 2021), and we describe our strategy for each dataset below.”
ExpertQA: Expert-Curated Questions and Attributed Answers (2023, background): “Several issues have been pointed out with conducting robust evaluation for long-form QA (Krishna et al., 2021).”
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models (2023, methodology): “On the other hand, conventional automated evaluation metrics (e.g., BLEU, ROUGE) cannot capture the depth and granularity of human evaluation (Papineni et al., 2002; Lin, 2004b; Zhang et al., 2019; Krishna et al., 2021).”
Evidentiality-guided Generation for Knowledge-Intensive NLP Tasks (2021, background): “Recent work reports similar issues in long-form QA evaluations (Krishna et al., 2021).”
GooAQ: Open Question Answering with Diverse Answer Types (2021, background): “They are even better than the same architectures trained with ELI5’s own training data (19.0 and 22.7, resp.) and on par with retrieval based state-of-the-art models (23.4).”
FeTaQA: Free-form Table Question Answering (2021, background): “By contrast, ELI5 (Fan et al., 2019) is a long form question answering dataset containing a diverse set of complex questions, each paired with a paragraph-long answer and 100 relevant web source documents (Petroni et al., 2020; Krishna et al., 2021).”