ELI5: Long Form Question Answering

By Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Aaron Edward Weston, Michael Auli. 2019

821 citations recorded by the source corpus.

We introduce the first large-scale corpus for long form question answering, a task requiring elaborate and in-depth answers to open-ended questions. The dataset comprises 270K threads from the Reddit forum “Explain Like I’m Five” (ELI5) where an online community provides answers to questions which are comprehensible by five year olds. Compared to existing datasets, ELI5 comprises diverse questions requiring multi-sentence answers. We provide a large set of web documents to help answer the question. Automatic and human evaluations show that an abstractive model trained with a multi-task objective outperforms conventional Seq2Seq, language modeling, as well as a strong extractive baseline.However, our best model is still far from human performance since raters prefer gold responses in over 86% of cases, leaving ample opportunity for future improvement.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “Generative question answering was mostly considered in previous work for datasets requiring to generate answers, such as NarrativeQA (Koˇcisk`y et al., 2018), CoQA (Reddy et al., 2019) or ELI5 (Fan et al., 2019).”
ReAct: Synergizing Reasoning and Acting in Language Models (2022, background): “, 2021) uses an LM to interact with web browsers, navigate through web pages, and infer answers to complicated questions from ELI5 (Fan et al., 2019).”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, background): “For example, a simple language model achieves the best ELI5 performance, but the worst SQUAD results.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “We train on 140 GB of text drawing from a wide variety of domains: Wikipedia (En, De, Es, Fr), Project Gutenberg 1 , submissions from 45 subreddits, OpenWebText 2 , a large collection of news data (Hermann et al., 2015; Barrault et al., 2019; Sandhaus, 2008; Grusky et al., 2018), Amazon Reviews (McA”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “ELI5 (Fan et al., 2019) 12.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, methodology): “, 2017c), ROUGE (Lin, 2004; Fan et al., 2019), CIDEr (Vedantam, Zitnick, and Parikh, 2015; Zhou et al.”
Autoregressive Entity Retrieval (2020, methodology): “…domain question answering using Natural Questions (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018c), TriviaQA (Joshi et al., 2017), ELI5 (Fan et al., 2019); slot ﬁlling with T-REx (Elsahar et al., 2018), Zero Shot RE (Levy et al., 2017); entity disambiguation on AIDA CoNLL-YAGO,…”
ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, neutral): “– KILT: FEVER (Thorne et al., 2018), AIDA (Yosef et al., 2011), WNED (Guo & Barbosa, 2014), T-REx (Guo & Barbosa, 2014), NQ (Kwiatkowski et al., 2019), HoPo (Yang et al., 2018), TQA (Joshi et al., 2017), ELI5 (Fan et al., 2019), WoW (Dinan et al., 2019).”
Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Multi-turn supported and plausible evaluation We assess Sparrow’s responses and accompanying evidence through human evaluation, using the metrics of supported and plausible as defined in section 2.4 and GopherCite (Menick et al., 2022).”
Survey of Hallucination in Natural Language Generation (2022, background): “Normally GQA system involves searching an external knowledge source for information relevant to the question.”
KILT: a Benchmark for Knowledge Intensive Language Tasks (2020, background): “Likewise, while the classical approach of information extraction for populating a Knowledge Base (KB, Riedel et al., 2013; Surdeanu and Ji, 2014) seems out-of-fashion, recent results show that they remain contenders (Fan et al., 2019a; Xiong et al., 2019).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, positive): “These tasks include recipe generation (Bie´n et al., 2020) from a given set of ingredients, story generation for a given prompt from Writing Prompts (WP) subreddit, and long form question answering, ELI5 (Fan et al., 2019).”
Reducing Transformer Depth on Demand with Structured Dropout (2019, methodology): “We follow the Transformer Big architecture and training procedure of Fan et al. (2019).”
Teaching language models to support answers with verified quotes (2022, methodology): “We evaluate using questions drawn from the Natural Questions (Kwiatkowski et al. (2019)) and ELI5 (Fan et al. (2019)) datasets.”
Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation (2024, background): “Retrieval-augmented generation (RAG) techniques (Lewis et al., 2020; Fan et al., 2019; Guu et al., 2020) have become a powerful approach by leveraging the strengths of retrieval systems and the generative capabilities of LLMs.”
Datasets: A Community Library for Natural Language Processing (2021, background): “co/datasets/ Figure 1: The data card for ELI5 (Fan et al., 2019).”