The TREC question answering track

By Ellen M. Voorhees. 2001

379 citations recorded by the source corpus.

The Text REtrieval Conference (TREC) question answering track is an effort to bring the benefits of large-scale evaluation to bear on a question answering (QA) task. The track has run twice so far, first in TREC-8 and again in TREC-9. In each case, the goal was to retrieve small snippets of text that contain the actual answer to a question rather than the document lists traditionally returned by text retrieval systems. The best performing systems were able to answer about 70% of the questions in TREC-8 and about 65% of the questions in TREC-9. While the 65% score is a slightly worse result than the TREC-8 scores in absolute terms, it represents a very significant improvement in question answering systems. The TREC-9 task was considerably harder than the TREC-8 task because TREC-9 used actual users’ questions while TREC-8 used questions constructed for the track. Future tracks will continue to challenge the QA community with more difficult, and more realistic, question answering tasks.

Subfields: Artificial Intelligence, Sociology and Political Science, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Sociology and Political Science

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, methodology): “Second, when fine-tuning on 128 examples only, we lower the batch size to 32 and run for 80 epochs.7 We use BM25 negatives produced by Karpukhin et al. (2020), and do not create hard negatives by the model itself (Xiong et al., 2021).”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, background): “Open-domain question answering (QA) has long considered retrieval as an intermediate step towards its solution (Voorhees, 2001), but has become a more intensively studied topic recently for neural models, first using simple vector-space based retrievers (Chen et al., 2017), and then more recently…”
Sparse, Dense, and Attentional Representations for Text Retrieval (2020, background): “Retrieving relevant documents is a core task for language technology, and is a component of applications such as information extraction and question answering (e.g., Narasimhan et al., 2016; Kwok et al., 2001; Voorhees, 2001).”
Coh-Metrix: Analysis of text on cohesion and language (2004, neutral): “These disciplines include computational linguistics (Allen, 1995; Jurafsky & Martin, 2000; Moore & Wiemer-Hastings, 2003), corpus linguistics (Biber, Conrad, & Reppen, 1998; Marcus, Santorini, & Marcinkiewicz, 1993), information extraction (DARPA, 1995; Lehnert, 1997; Pennebaker & Francis, 1999), in”
Domain-agnostic Question-Answering with Adversarial Training (2019, background): “…6 out-of-domain undisclosed private test datasets, which are BioProcess (BP) (Scaria et al., 2013), ComplexWebQuestion (CQ) (Talmor and Berant, 2018), MCTest (MC) (Richardson et al., 2013), QAMR (MR) (Michael et al., 2017), QAST (ST) (Jitkrittum et al., 2009) and TREC (TR) (Voorhees, 2001).”
A Survey of Paraphrasing and Textual Entailment Methods (2009, background): “In QA systems for document collections (Voorhees, 2001; Pasca, 2003; Harabagiu & Moldovan, 2003; Mollá & Vicedo, 2007), a question may be phrased differently than in a document that contains the answer, and taking such variations into account can improve system performance significantly…”
AutoTutor: A tutor with dialogue in natural language (2004, background): “…and dialogue analyzers that are capable of rapidly extracting information from naturalistic text for information retrieval, machine translation, and speech recognition (Allen, 1995; DARPA, 1995; Harabagiu, Maiorano, & Pasca, 2002; Jurafsky & Martin, 2000; Manning & Schütze, 1999; Voorhees, 2001).”
CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning (2021, background): “Passage retrieval in an open-domain conversational question answering (CQA) system (Anantha et al., 2021), compared to standard retrieval tasks (Voorhees and Tice, 2000; Bajaj et al., 2016), poses new challenges of understanding user questions within the dialogue context.”
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering (2020, methodology): “This line of work (which we will refer to as TextQA) has been popularized by the TREC QA tracks (Voorhees and Tice, 2000), and has seen explosive growth with the advent of neural machine reading (MRC) (Rajpurkar et al., 2016) models.”
FCM: A Fine-grained Comparison Model for Multi-turn Dialogue Reasoning (2021, neutral): “On MuTual, we display the recall and Mean Reciprocal Rank measures(Voorhees and Tice, 2000), i.”
A general approximation framework for direct optimization of information retrieval measures (2010, methodology): “1 IR Measures To evaluate the effectiveness of a ranking model, IR measures such as Precision, AP (Average Precision) [1], NDCG (Normalized Discounted Cumulative Gain) [13] and MRR (Mean Reciprocal Rank) [21] are being used.”
Effects of Model Choice, Corpus Context, and Post Hoc Correction on Layer-Level Embedding Degradation in Clinical Document Retrieval: Experimental Study (2026, unknown): “Clinical information retrieval evaluation has emphasized known-item retrieval [27] and the impact of query-document lexical overlap on apparent performance [23]; both inform the lexical-overlap audit reported below.”