TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions

By Qiang Ning, Hao Wu, Rujun Han, Nanyun Peng, Matt Gardner, Dan Roth. 2020

128 citations recorded by the source corpus.

A critical part of reading is being able to understand the temporal relationships between events described in a passage of text, even when those relationships are not explicitly stated. However, current machine reading comprehension benchmarks have practically no questions that test temporal phenomena, so systems trained on these benchmarks have no capacity to answer questions such as "what happened before/after [some event]?" We introduce TORQUE, a new English reading comprehension benchmark built on 3.2k news snippets with 21k human-generated questions querying temporal relationships. Results show that RoBERTa-large achieves an exact-match score of 51% on the test set of TORQUE, about 30% behind human performance.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Additionally, we did not test the ability to model temporal expressions such as “before” or “during”, and we did not investigate temporal commonsense (e.g., Zhou et al. 2019), temporal ordering (e.g., Ning et al. 2020) events (e.g., Zhou et al. 2021).”
ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, methodology): “TORQUE (Ning et al., 2020) is an MRC/QA dataset where annotators first identify event triggers in given passages and then ask questions regarding event temporal relations (ordering).”
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data (2020, background): “Our task differs from that of other works that require an understanding of temporal relationships (Ning et al., 2020) and temporal commonsense reasoning (Zhou et al., 2019), as our task forces a model to make a forecasting judgement.”
Extracting Temporal Event Relation with Syntactic-Guided Temporal Graph Transformer (2021, background): “…developed over the past years (Puste-jovsky et al., 2003; UzZaman et al., 2013; Cas-sidy et al., 2014b; Ning et al., 2018; Vashishtha et al., 2019; Ning et al., 2020), among which, MA-TRES and TimeBank-Dense(TB-Dense) are the two largest manually annotated temporal relation datasets, which only…”
Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation (2022, methodology): “To measure the inter-annotator agreement (IAA), we use the Worker Agreement With Aggregate (WAWA) score, following Ning et al. (2020) and Sheng et al. (2021).”
Natural Language Processing with Commonsense Knowledge: A Survey (2021, background): “• T ORQUE [83] is a reading comprehension dataset specifically designed to evaluate a model’s ability to understand temporal relationships between events in text.”
Question Answering Over Temporal Knowledge Graphs (2021, background): “One such dataset is TORQUE (Ning et al., 2020), where the system is given a question along with some context (a text passage) and is asked to answer a multiple choice question with five choices.”
Reasoning with Transformer-based Models: Deep Learning, but Shallow Reasoning (2021, methodology): “The TORQUE dataset [Ning et al., 2020] asks temporal relation questions such as which events have already ﬁnished, given a short passage of text.”
What Ingredients Make for an Effective Crowdsourcing Protocol for Difficult NLU Data Collection Tasks? (2021, background): “Another commonly used strategy when crowd-sourcing NLP datasets is to only qualify workers who pass an initial quiz or perform well in prelim-inary crowdsourcing batches (Wang et al., 2013; Cotterell and Callison-Burch, 2014; Ning et al., 2020; Shapira et al., 2020; Roit et al., 2020).”
Temporal Reasoning on Implicit Events from Distant Supervision (2020, methodology): “All steps are implemented with the CrowdAQ platform (Ning et al., 2020a) with qualiﬁcation exams.”
“Nice Try, Kiddo”: Investigating Ad Hominems in Dialogue Responses (2021, methodology): “To measure inter-annotator agreement, we calculate the Worker Agreement With Aggregate (WAWA) score, following Ning et al. (2020).”