SemEval-2015 Task 6: Clinical TempEval

By Steven Bethard, Leon Derczynski, Guergana K. Savova, James Pustejovsky, Marc Verhagen. 2015

166 citations recorded by the source corpus.

Clinical TempEval 2015 brought the temporal information extraction tasks of past TempEval campaigns to the clinical domain. Nine sub-tasks were included, covering problems in time expression identification, event expression identification and temporal relation identification. Participant systems were trained and evaluated on a corpus of clinical notes and pathology reports from the Mayo Clinic, annotated with an extension of TimeML for the clinical domain. Three teams submitted a total of 13 system runs, with the best systems achieving near-human performance on identifying events and times, but with a large performance gap still remaining for temporal relations.

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, background): “…crucial for natural language understanding, and facilitates many real-world applications, such as tracking biomedical histories (Sun et al., 2013; Bethard et al., 2015, 2016, 2017), generating stories (Yao et al., 2019; Goldfarb-Tarrant et al., 2020), and forecasting social events (Li et al.,…”
Domain Knowledge Empowered Structured Neural Net for End-to-End Event Temporal Relation Extraction (2020, unknown): “Following this early effort, a series of clinical event temporal relation challenges were created in the following years ((Bethard et al., 2015, 2016, 2017)).”
Clinical Natural Language Processing in languages other than English: opportunities and challenges (2018, methodology): “However, there are currently no sharable clinical datasets comparable to the i2b2 datasets [142, 143], the ShARe corpus [144], the THYME corpus [145, 146] or the MIMIC corpus [147] in languages other than English except the Turku Clinical TreeBank and PropBank [32, 38, 148] in Finnish and the small…”
Recent Advances in Clinical Natural Language Processing in Support of Semantic Analysis (2015, methodology): “The THYME corpus was released as part of the SemEval-2015 Task 6: Clinical TempEval challenge [65], where the system approach results met or were close to human agreement on all subtasks except temporal relationships using machine learning approaches [66].”
Clinical Text Data in Machine Learning: Systematic Review (2020, background): “[60] Times, events, and temporal relations among them 600 Clinical notes, pathology reports Mayo 2015 SemEval/THYME [129]”
Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances (2018, unknown): “Typically, in evaluating clinical NLP methods, a gold standard corpus with instance annotations is developed, and used to measure whether or not an NLP approach correctly identifies and classifies these instances.”
Clinical Temporal Relation Extraction with Probabilistic Soft Logic Regularization and Global Inference (2020, unknown): “I2b2-2012 (Sun, Rumshisky, and Uzuner 2013) and Clinical TempEval (Bethard et al. 2015, 2016, 2017) are some great efforts of building clinical datasets with extensive annotations including labels of clinical events and temporal relations, the second of which was not tested in our paper due to lack ”
A Multi-Axis Annotation Scheme for Event Temporal Relations (2018, background): “…for event understanding, and it has drawn much attention in the natural language processing (NLP) community recently (UzZaman et al., 2013; Chambers et al., 2014; Llorens et al., 2015; Minard et al., 2015; Bethard et al., 2015, 2016, 2017; Leeuwenberg and Moens, 2017; Ning et al., 2017, 2018a,b).”
Neural Temporal Relation Extraction (2017, methodology): “The vast majority of systems in temporal information extraction challenges, such as the i2b2 (Sun et al., 2013) and Clinical TempEval tasks (Bethard et al., 2015; Bethard et al., 2016), used classiﬁers with a large number of manually engineered features.”
SemEval-2017 Task 12: Clinical TempEval (2017, background): “In the Clinical TempEval shared tasks (Bethard et al., 2015, 2016), participant systems have competed to identify critical components of the timeline of a clinical text: time expressions, event expressions, and temporal relations.”