Building a question answering test collection

By Ellen M. Voorhees, Dawn M. Tice. 2000

649 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, background): “A standard task for open domain IR-style QA is the annual TREC competitions (Voorhees and Tice, 2000), which contains questions from various domains but is limited in size.”
SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016, background): “The annual evaluations at the Text REtreival Conference (TREC) (Voorhees and Tice, 2000) led to many advances in open-domain QA, many of which were used in IBM Watson for Jeopardy! (Ferrucci et al.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “We use a set of binary classiﬁcation tasks (see Table 1) that covers various types of sentence classiﬁcation, including sentiment analysis (MR and both binary and ﬁne-grained SST) (Pang and Lee, 2005; Socher et al., 2013), question-type (TREC) (Voorhees and Tice, 2000), product reviews (CR) (Hu and ”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “from generating an answer word-by-word (Mitra, 2017) or finding an answer from within an entire corpus as in TREC (Voorhees and Tice, 2000) and DrQA (Chen et al.”
Key-Value Memory Networks for Directly Reading Documents (2016, methodology): “Question answering (QA) has been a long standing research problem in natural language processing, with the first systems attempting to answer questions by directly reading documents (Voorhees and Tice, 2000).”
Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, methodology): “Reduces Drop from 0-shot to 1-shot For the baseline, there are four cases where there is a drop in accuracy when We plot the difference in standard deviation between contextual calibration and the baseline from Table 1. moving from 0-shot to 1-shot (TREC, AGNews, DBpedia, SST-2).”
Self-training Improves Pre-training for Natural Language Understanding (2020, background): “, 2013) binary and fine-grained sentiment analysis datasets (SST-2 and SST-5), on product classification (CR) from (Hu and Liu, 2004), hate-speech comment classification3 (IMP), question classification (TREC) from (Voorhees and Tice, 2000) and named entity recognition (CoNLL 2002) from (Sang and De ”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “Historically, many of the question answering datasets have only thousands of question answering pairs, such as WebQues-tions (Berant et al., 2013), MCTest (Richardson et al., 2013), WikiQA (Yang et al., 2015), and TREC-QA (Voorhees and Tice, 2000).”
Achieving Human Parity on Visual Question Answering (2021, background): “It was not until the release of QA track evaluations of the Text REtrieval Conference (TREC) [76] that text-based QA received gradually increasing attention.”
Quasar: Datasets for Question Answering by Search and Reading (2017, methodology): “Open-Domain QA: Early research into open-domain QA was driven by the TREC-QA challenges organized by the National Institute of Standards and Technology (NIST) (Voorhees and Tice, 2000).”
Open Question Answering with Weakly Supervised Embedding Models (2014, background): “Large-scale question answering has a long history, mostly initiated via the TREC tracks [22].”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “from the TREC QA tasks (Voorhees and Tice, 2000) from 1999-2002.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, methodology): “We also experiment with post-processing methods (BERT- 16https://github.com/facebookresearch/ SentEval Model MR CR SUBJ MPQA SST TREC MRPC Avg.”