TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension
By Mandar Joshi, Eunsol Choi, Daniel S. Weld, Luke S. Zettlemoyer. 2017
4,054 citations recorded by the source corpus.
We present TriviaQA, a challenging reading comprehension dataset containing over 650K question-answer-evidence triples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts and independently gathered evidence documents, six per question on average, that provide high quality distant supervision for answering the questions. We show that, in comparison to other recently introduced large-scale datasets, TriviaQA (1) has relatively complex, compositional questions, (2) has considerable syntactic and lexical variability between questions and corresponding answer-evidence sentences, and (3) requires more cross sentence reasoning to find answers. We also present two baseline algorithms: a feature-based classifier and a state-of-the-art neural network, that performs well on SQuAD reading comprehension. Neither approach comes close to human performance (23% and 40% vs. 80%), suggesting that TriviaQA is a challenging testbed that is worth significant future study.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Yejin Choi
- Hannaneh Hajishirzi
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Jitendra Malik
- Ming-Hsuan Yang
- Mohit Bansal
- Andrew Zisserman
In Information Retrieval & Search
- Jimmy J. Lin
- Hannaneh Hajishirzi
- Jamie Callan
- Wen-tau Yih
- Sebastian Riedel
- Danqi Chen
- Luyu Gao
- Gautier Izacard
In Evaluation & Benchmarks
- Mohit Bansal
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Douwe Kiela
- Ashish Sabharwal
- Yejin Choi
- Samuel R. Bowman
- Matt Gardner
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “We therefore use very modest data augmentation in our submitted system by jointly training on SQuAD and TriviaQA (Joshi et al., 2017).”
- Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “We evaluate our model on the task of question answering using recently released SQuAD and TriviaQA Wikipedia (Joshi et al., 2017), which have gained a huge attention over the past year.”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.”
- Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “Pipelined approaches select a single paragraph ∗ Work completed while interning at the Allen Institute for Artificial Intelligence from the input documents, which is then passed to the paragraph model to extract an answer (Joshi et al., 2017; Wang et al., 2017a).”
- Language Models are Few-Shot Learners (2020, methodology): “On TriviaQA [JCWZ17], GPT-3 zero-shot already outperforms the fine-tuned T5-11B by 14.2%, and also outperforms a version with Q&A tailored span prediction during pre-training by 3.8%.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “We note that when results are reported on the test set, they are typically based on an ensemble of models and/or leverage external datasets (e.g. TriviaQA [Joshi et al., 2017] or NewsQA [Trischler et al., 2016]) to augment the small SQuAD training set.”
- Training Compute-Optimal Large Language Models (2022, positive): “, 2019) and TriviaQA (Joshi et al., 2017), Chinchilla outperforms Gopher in all cases.”
- PaLM: Scaling Language Modeling with Pathways (2022, background): “The benchmarks include: • Open-Domain Closed-Book Question Answering tasks:5 TriviaQA (Joshi et al., 2017), Natural Questions (Kwiatkowski et al., 2019), Web Questions (Berant et al., 2013) • Cloze and Completion tasks: LAMBADA (Paperno et al., 2016), HellaSwag (Zellers et al., 2019), StoryCloze (Mo”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “In Table 4, we report performance on NaturalQuestions, and in Table 5, we report on TriviaQA.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We evaluate the Llama 2 model together with other open-source models on the NaturalQuestions and TriviaQA benchmarks (Table 22).”
- Mistral 7B (2023, neutral): “…PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 • Code: Humaneval…”
- Mixtral of Experts (2024, neutral): “…PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 • Code: Humaneval…”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Our RAG models achieve state-of-the-art results on open Natural Questions [25], WebQuestions [3] and CuratedTrec [2] and strongly outperform recent approaches that use specialised pre-training objectives on TriviaQA [20].”
- Dense Passage Retrieval for Open-Domain Question Answering (2020, background): “TriviaQA (Trivia) (Joshi et al., 2017) contains a set of trivia questions with answers that were originally scraped from the Web.”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “• TriviaQA (Joshi et al., 2017) contains questions gathered from trivia and quiz-league websites.”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “…evaluate our model on the following tasks and datasets included in KILT: question answering: NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018); slot filling: Zero Shot RE (Levy et al., 2017) and T-REx (Elsahar et al., 2018); entity linking:…”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “Question Answering We look at Web Questions (Berant et al., 2013), Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “, 2008); and TriviaQA (Joshi et al., 2017), a collection of questions from quiz league websites where each question is accompanied by pages from web and Wikipedia searches that may contain the answer.”
- NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “In recent years, researchers have published a large number of annotated MRC datasets such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016), RACE (Lai et al., 2017), Trivi-aQA (Joshi et al., 2017) and so on.”