SemEval-2017 Task 12: Clinical TempEval

By Steven Bethard, Guergana K. Savova, Martha Palmer, James Pustejovsky. 2017

111 citations recorded by the source corpus.

Clinical TempEval 2017 aimed to answer the question: how well do systems trained on annotated timelines for one medical condition (colon cancer) perform in predicting timelines on another medical condition (brain cancer)? Nine sub-tasks were included, covering problems in time expression identification, event expression identification and temporal relation identification. Participant systems were evaluated on clinical and pathology notes from Mayo Clinic cancer patients, annotated with an extension of TimeML for the clinical domain. 11 teams participated in the tasks, with the best systems achieving F1 scores above 0.55 for time expressions, above 0.70 for event expressions, and above 0.40 for temporal relations. Most tasks observed about a 20 point drop over Clinical TempEval 2016, where systems were trained and evaluated on the same domain (colon cancer).

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nanyun Peng
2. Xiang Ren
3. Rujun Han
4. Dan Roth
5. Qiang Ning
6. Matt Gardner
7. Hannaneh Hajishirzi
8. Yejin Choi

In Molecular Biology

1. Xiang Ren
2. Dan Roth
3. Hao Wu
4. Nanyun Peng
5. Qiang Ning
6. Matt Gardner
7. Rujun Han

Citations in the corpus

• ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoning (2020, background): “…crucial for natural language understanding, and facilitates many real-world applications, such as tracking biomedical histories (Sun et al., 2013; Bethard et al., 2015, 2016, 2017), generating stories (Yao et al., 2019; Goldfarb-Tarrant et al., 2020), and forecasting social events (Li et al.,…”
• TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, background): “Time is important for understanding events and stories described in natural language text such as news articles, social media, financial reports, and electronic health records (Verhagen et al., 2007, 2010; UzZaman et al., 2013; Minard et al., 2015; Bethard et al., 2016, 2017; Laparra et al., 2018).”
• A Multi-Axis Annotation Scheme for Event Temporal Relations (2018, background): “…for event understanding, and it has drawn much attention in the natural language processing (NLP) community recently (UzZaman et al., 2013; Chambers et al., 2014; Llorens et al., 2015; Minard et al., 2015; Bethard et al., 2015, 2016, 2017; Leeuwenberg and Moens, 2017; Ning et al., 2017, 2018a,b).”
• Domain Knowledge Empowered Structured Neural Net for End-to-End Event Temporal Relation Extraction (2020, unknown): “Following this early effort, a series of clinical event temporal relation challenges were created in the following years ((Bethard et al., 2015, 2016, 2017)).”