SemEval-2016 Task 12: Clinical TempEval

By Steven Bethard, Guergana K. Savova, Wei-Te Chen, Leon Derczynski, James Pustejovsky, Marc Verhagen. 2016

187 citations recorded by the source corpus.

Clinical TempEval 2016 evaluated temporal information extraction systems on the clinical domain. Nine sub-tasks were included, covering problems in time expression identification, event expression identification and temporal relation identification. Participant systems were trained and evaluated on a corpus of clinical and pathology notes from the Mayo Clinic, annotated with an extension of TimeML for the clinical domain. 14 teams submitted a total of 40 system runs, with the best systems achieving near-human performance on identifying events and times. On identifying temporal relations, there was a gap between the best systems and human performance, but the gap was less than half the gap of Clinical TempEval 2015.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Qiang Ning
2. Daniel Khashabi
3. Dan Roth
4. Xiang Ren
5. Rujun Han
6. Ben Zhou
7. Matt Gardner
8. Tushar Khot

Citations in the corpus

• ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, background): “…crucial for natural language understanding, and facilitates many real-world applications, such as tracking biomedical histories (Sun et al., 2013; Bethard et al., 2015, 2016, 2017), generating stories (Yao et al., 2019; Goldfarb-Tarrant et al., 2020), and forecasting social events (Li et al.,…”
• Extracting Temporal Event Relation with Syntactic-Guided Temporal Graph Transformer (2021, methodology): “…model it as a pairwise classification problem (Mani et al., 2006a; Verhagen et al., 2007; Verhagen and Pustejovsky, 2008; Verhagen et al., 2010; Bethard et al., 2016; MacAvaney et al., 2017) and use statistical machine learning techniques with hand-crafted features and carefully designed…”
• Representations of Time Expressions for Temporal Relation Extraction with Convolutional Neural Networks (2017, methodology): “We followed the evaluation setup of Clinical Temp-Eval 2016 (Bethard et al., 2016).”
• A Multi-Axis Annotation Scheme for Event Temporal Relations (2018, background): “…for event understanding, and it has drawn much attention in the natural language processing (NLP) community recently (UzZaman et al., 2013; Chambers et al., 2014; Llorens et al., 2015; Minard et al., 2015; Bethard et al., 2015, 2016, 2017; Leeuwenberg and Moens, 2017; Ning et al., 2017, 2018a,b).”
• Using clinical Natural Language Processing for health outcomes research: Overview and actionable suggestions for future advances (2018, unknown): “These include the Informatics for Integrating Biology and the Bedside (i2b2) challenges [5–9], the Conference and Labs of the Evaluation Forum (CLEF) eHealth challenges [10–13], and the Semantic Evaluation (SemEval) challenges [14–16].”
• TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, background): “Time is important for understanding events and stories described in natural language text such as news articles, social media, financial reports, and electronic health records (Verhagen et al., 2007, 2010; UzZaman et al., 2013; Minard et al., 2015; Bethard et al., 2016, 2017; Laparra et al., 2018).”
• Domain Knowledge Empowered Structured Neural Net for End-to-End Event Temporal Relation Extraction (2020, unknown): “Following this early effort, a series of clinical event temporal relation challenges were created in the following years ((Bethard et al., 2015, 2016, 2017)).”
• Neural Temporal Relation Extraction (2017, methodology): “The vast majority of systems in temporal information extraction challenges, such as the i2b2 (Sun et al., 2013) and Clinical TempEval tasks (Bethard et al., 2015; Bethard et al., 2016), used classifiers with a large number of manually engineered features.”
• Temporal Common Sense Acquisition with Minimal Supervision (2020, background): “inference, and QA (UzZaman et al., 2013; Chambers et al., 2014; Llorens et al., 2015; Bethard et al., 2016; Leeuwenberg and Moens, 2017; Ning et al., 2018b), rely on understanding time.”
• SemEval-2017 Task 12: Clinical TempEval (2017, background): “In the Clinical TempEval shared tasks (Bethard et al., 2015, 2016), participant systems have competed to identify critical components of the timeline of a clinical text: time expressions, event expressions, and temporal relations.”
• A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, methodology): “In Haralabopoulos et al. (2020) applied both voting and stacking based on several deep learning models, namely LSTM, GRU, CNN, RCNN, and DNN, using two English tweets datasets (SemEval (Bethard et al., 2016), Toxic Comment (van Aken et al., 2018)).”
• Clinical Temporal Relation Extraction with Probabilistic Soft Logic Regularization and Global Inference (2020, unknown): “I2b2-2012 (Sun, Rumshisky, and Uzuner 2013) and Clinical TempEval (Bethard et al. 2015, 2016, 2017) are some great efforts of building clinical datasets with extensive annotations including labels of clinical events and temporal relations, the second of which was not tested in our paper due to lack ”
• Enhancing Clinical Concept Extraction with Contextual Embedding (2019, background): “…those from i2b2 (Uzuner et al., 2011; Sun et al., 2013; Stubbs et al., 2015), ShARe/CLEF (Suominen et al., 2013; Kelly et al., 2014), and SemEval (Pradhan et al., 2014; El-hadad et al., 2015; Bethard et al., 2016) act as evaluation benchmarks and datasets for training machine learning (ML) models.”