A Multi-Axis Annotation Scheme for Event Temporal Relations

By Qiang Ning, Hao Wu, Dan Roth. 2018

197 citations recorded by the source corpus.

Existing temporal relation (TempRel) annotation schemes often have low inter-annotator agreements (IAA) even between experts, suggesting that the current annotation task needs a better definition. This paper proposes a new multi-axis modeling to better capture the temporal structure of events. In addition, we identify that event end-points are a major source of confusion in annotation, so we also propose to annotate TempRels based on start-points only. A pilot expert annotation effort using the proposed scheme shows significant improvement in IAA from the conventional 60’s to 80’s (Cohen’s Kappa). This better-defined annotation scheme further enables the use of crowdsourcing to alleviate the labor intensity for each annotator. We hope that this work can foster more interesting studies towards event understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Signal Processing

Citations in the corpus

Tailor: Generating and Perturbing Text with Semantic Controls (2021, background): “See §B for more details. tasks: boolean question answering (BoolQ: Clark et al., 2019), extractive QA (SQuAD: Rajpurkar et al., 2016), dependency tree parsing (UD English: Nivre et al., 2016), and temporal relation extraction (MATRES: Ning et al., 2018).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “…datasets (full descriptions are provided in Section A): • NLVR2 (Suhr et al., 2019) • IMDb sentiment analysis (Maas et al., 2011) • MATRES Temporal RE (Ning et al., 2018) • English UD parsing (Nivre et al., 2016) • PERSPECTRUM (Chen et al., 2019) • DROP (Dua et al., 2019) • Quoref (Dasigi et…”
ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, methodology): “In this work, we study two prominent event temporal reasoning tasks as shown in Figure 1: event relation extraction (ERE) (Chambers et al., 2014; Ning et al., 2018; O’Gorman et al., 2016; Mostafazadeh et al., 2016) that predicts temporal relations between a pair of events, and machine reading…”
“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding (2019, background): “…work has investigated some of these aspects, either explicitly or implicitly (e.g., duration (Gusev et al., 2011; Williams, 2012) and ordering (Chklovski and Pantel, 2004; Ning et al., 2018b)), but none of them have deﬁned or studied all aspects of temporal commonsense in a uniﬁed framework.”
Evaluating NLP Models via Contrast Sets (2020, background): “…datasets (full descriptions are provided in Section A): • NLVR2 (Suhr et al., 2019) • IMDb sentiment analysis (Maas et al., 2011) • MATRES Temporal RE (Ning et al., 2018) • English UD parsing (Nivre et al., 2016) • PERSPECTRUM (Chen et al., 2019) • DROP (Dua et al., 2019) • Quoref (Dasigi et…”
Effective Distant Supervision for Temporal Relation Extraction (2020, methodology): “However, even simplified annotation schemes designed for crowdsourcing (Ning et al., 2018b; Vashishtha et al., 2019) can struggle to acquire high-accuracy judgments about nebulous phenomena, leading to a scarcity of high-quality labeled data.”
Joint Reasoning for Temporal and Causal Relations (2018, methodology): “For more details about this data collection scheme, please refer to (Ning et al., 2018b) for more details.”
Deep Structured Neural Network for Event Temporal Relation Extraction (2019, methodology): “This constraint speciﬁes that if event i causes event j, then i has to occur before j. Note that this constraint only has 91.9% accuracy in TCR data (Ning et al., 2018a), but it can help improve model performance based on our experiments.”
Temporal Reasoning on Implicit Events from Distant Supervision (2020, methodology): “We show that our model improves over T RACIE and MATRES, suggesting the effectiveness of high-precision pre-training and symbolic temporal reasoning.”
Partial Or Complete, That’s The Question (2019, unknown): “Following CogComp-Time (Ning et al., 2018d), we choose the same features and sparse-averaged perceptron algorithm as the L EARN component and ILP as I NFERENCE for SSPAN.”
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, methodology): “Research on temporal relations often focuses on algorithmic improvement, such as structured inference (Do et al., 2012; Cham-bers et al., 2014; Ning et al., 2018a), structured learning (Leeuwenberg and Moens, 2017; Ning et al., 2017), and neural networks (Dligach et al., 2017; Lin et al., 2017;…”
Extracting Temporal Event Relation with Syntactic-Guided Temporal Graph Transformer (2021, methodology): “…relation extraction benchmarks have been developed over the past years (Puste-jovsky et al., 2003; UzZaman et al., 2013; Cas-sidy et al., 2014b; Ning et al., 2018; Vashishtha et al., 2019; Ning et al., 2020), among which, MA-TRES and TimeBank-Dense(TB-Dense) are the two largest manually…”
Contextualized Word Embeddings Enhanced Event Temporal Relation Extraction for Story Understanding (2019, background): “Despite multiple productive research threads on temporal and causal relation modeling among events (Chambers et al., 2014; Ning et al., 2018a; Meng and Rumshisky, 2018) and event relation annotation for story understanding (Mostafazadeh et al., 2016b), the intersection of these two threads seems…”
Joint Constrained Learning for Event-Event Relation Extraction (2020, methodology): “We also incorporate commonsense knowledge from T EM P ROB (Ning et al., 2018a).”
Joint Event and Temporal Relation Extraction with Shared Representations and Structured Prediction (2019, methodology): “Recent data construction efforts such as MATRES (Ning et al., 2018a) further enhance the data quality by using a multi-axis annotation scheme and adopting a startpoint of events to improve inter-annotator agreements.”