MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
By Daniel Fernando Campos, Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, Li Deng, Bhaskar Mitra. 2016
3,491 citations recorded by the source corpus.
This paper presents our recent work on the design and development of a new, large scale dataset, which we name MS MARCO, for MAchine Reading COmprehension. This new dataset is aimed to overcome a number of well-known weaknesses of previous publicly available datasets for the same task of reading comprehension and question answering. In MS MARCO, all questions are sampled from real anonymized user queries. The context passages, from which answers in the dataset are derived, are extracted from real web documents using the most advanced version of the Bing search engine. The answers to the queries are human generated. Finally, a subset of these queries has multiple answers. We aim to release one million queries and the corresponding answers in the dataset, which, to the best of our knowledge, is the most comprehensive real-world dataset of its kind in both quantity and quality. We are currently releasing 100,000 queries with their corresponding answers to inspire work in reading comprehension and question answering along with gathering feedback from the research community.
Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
- Graham Neubig
- Yejin Choi
- Kyunghyun Cho
In Information Systems
- Jimmy J. Lin
- Rodrigo Nogueira
- Jamie Callan
- Sebastian Hofstätter
- Zeynep Akkalyoncu Yilmaz
- Zhuyun Dai
- Allan Hanbury
- Andrew Yates
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Wen-tau Yih
- Sebastian Hofstätter
- Luyu Gao
- Sebastian Riedel
- Hannaneh Hajishirzi
- Rodrigo Nogueira
In Evaluation & Benchmarks
- Mohit Bansal
- Hannaneh Hajishirzi
- Douwe Kiela
- Luke S. Zettlemoyer
- Matt Gardner
- Sebastian Riedel
- Robin Jia
- Jonathan Berant
Citations in the corpus
- Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “Additional datasets including SQuAD (Ra-jpurkar et al., 2016), WikiReading (Hewlett et al., 2016), MS Marco (Nguyen et al., 2016) and Triv-iaQA (Joshi et al., 2017) provided more realistic questions.”
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “The recently released MS Marco dataset (Nguyen et al., 2016) also contains independently authored questions and documents drawn from the search results.”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “For knowledge-intensive generation, we experiment with MS-MARCO [1] and Jeopardy question generation, and we find that our models generate responses that are more factual, specific, and diverse than a BART baseline.”
- Natural Questions: A Benchmark for Question Answering Research (2019, methodology): “MS Marco contains 100,000 questions with free-form answers.”
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “the MS Marco corpus (Nguyen et al., 2016).”
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
- CoQA: A Conversational Question Answering Challenge (2018, methodology): “…a hand-written grammar to create artificial questions (Weston et al., 2016; Welbl et al., 2018), paraphrasing artificial questions to natural questions (Saha et al., 2018; Talmor and Berant, 2018), or, in our case, letting humans ask natural questions (Rajpurkar et al., 2016; Nguyen et al., 2016).”
- Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
- Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, background): “Other datasets (e.g MS-MARCO and SQuAD) represent answers as a contiguous substring within the passage.”
- RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, methodology): “Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
- Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, background): “Machine reading comprehension (MRC) is a fundamental and long-standing goal of natural language understanding which aims to teach machine to answer question automatically according to given passage (Hermann et al. 2015; Ra-jpurkar et al. 2016; Nguyen et al. 2016; Zhang et al. 2018).”
- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “Recognizing the need to align research with the impact it will have on real users, the community has responded with datasets of information-seeking questions such as WikiQA (Yang et al., 2015), MS MARCO (Nguyen et al., 2016), QuAC Pronounced tie dye Q. A. —like the colorful t-shirt.”
- CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, background): “Work on Question Answering (QA) has mostly focused on answering factoid questions, where the answer can be found in a given context with little need for commonsense knowledge (Hermann et al., 2015; Rajpurkar et al., 2016; Nguyen et al., 2016; Joshi et al., 2017).”
- Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Questionanswering, which can be thought of as a single turn task is similar, either requiring reading comprehension (Rajpurkar et al., 2016) or retrieval as well in the more realistic case (Nguyen et al., 2016; Chen et al., 2017).”
- PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, methodology): “The MARCO benchmark (Nguyen et al., 2016) released by Microsoft is a good fit for evaluating generative QA models.”
- ELI5: Long Form Question Answering (2019, background): “MS MARCO (Nguyen et al., 2016), a dataset of crowdsourced responses to Bing queries, has written answers around 1 sentence long with short input passages.”
- Passage Re-ranking with BERT (2019, methodology): “In this paper, we describe in detail how we have re-purposed BERT as a passage re-ranker and achieved state-of-the-art results on the MS MARCO passage re-ranking task.”