TempQuestions: A Benchmark for Temporal Question Answering

By Jia Zhen, Abdalghani Abujabal, Rishiraj Saha Roy, Jannik Strotgen, Gerhard Weikum. 2018

135 citations recorded by the source corpus.

Answering complex questions is one of the challenges that question-answering (QA) systems face today. While complexity has several facets, question dimensions like temporal and spatial intents necessitate specialized treatment. Methods geared towards such questions need benchmarks that reflect the desired aspects and challenges. Here, we take a key step in this direction, and release a new benchmark, TempQuestions, containing 1,271 questions, that are all temporal in nature, paired with their answers. As a key contribution that enabled the creation of this resource, we provide a crisp definition for temporal questions. Most questions require decomposing them into sub-questions, and the questions are of a kind that they would be best evaluated on a combination of structured data and unstructured text sources. Experiments with two QA systems demonstrate the need for further research on complex questions.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Question Answering & Reading Comprehension

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, neutral): “T EMP LAMA is similar in spirit to KB-QA benchmarks which focus on temporal reasoning such as TempQuestions (Jia et al., 2018) and Cron-Questions (Saxena et al., 2021).”
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data (2020, background): “One pertinent example of a reasoning type that QA tasks test is the understanding of temporal and casual relations (Jia et al., 2018a,b; Sun et al., 2018; Ning et al., 2020).”
Question Answering Over Temporal Knowledge Graphs (2021, methodology): “TempQuestions (Jia et al., 2018a) is a KGQA dataset specifically aimed at temporal QA.”
Rethinking with Retrieval: Faithful Large Language Model Inference (2022, methodology): “In this experiment, we use the TempQuestions dataset (Jia et al., 2018) to investigate temporal reasoning.”
RealTime QA: What's the Answer Right Now? (2022, background): “Several recent works (Jia et al., 2018; Chen et al., 2021; Figure 1: R EAL T IME QA establishes a framework to benchmark question answering at the present time: answers (e.g., the number of Shohei Ohtani’s home runs) change in real time.”