Inferring Which Medical Treatments Work from Reports of Clinical Trials

By Eric P. Lehman, Jay DeYoung, Regina Barzilay, Byron C. Wallace. 2019

132 citations recorded by the source corpus.

How do we know if a particular medical treatment actually works? Ideally one would consult all available evidence from relevant clinical trials. Unfortunately, such results are primarily disseminated in natural language scientific articles, imposing substantial burden on those trying to make sense of them. In this paper, we present a new task and corpus for making this unstructured published scientific evidence actionable. The task entails inferring reported findings from a full-text article describing randomized controlled trials (RCT) with respect to a given intervention, comparator, and outcome of interest, e.g., inferring if a given article provides evidence supporting the use of aspirin to reduce risk of stroke, as compared to placebo. We present a new corpus for this task comprising 10,000+ prompts coupled with full-text articles describing RCTs. Results using a suite of baseline models — ranging from heuristic (rule-based) approaches to attentive neural architectures — demonstrate the difficulty of the task, which we believe largely owes to the lengthy, technical input texts. To facilitate further work on this important, challenging problem we make the corpus, documentation, a website and leaderboard, and all source code for baselines and evaluation publicly available.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Measuring Association Between Labels and Free-Text Rationales (2020, background): “…CommonsenseQA (Tal-mor et al., 2019), SNLI (Bowman et al., 2015), SST (Socher et al., 2013), AgNews (Del Corso et al., 2005), Evidence Inference (Lehman et al., 2019), Movie Reviews (Zaidan and Eisner, 2008), MultiRC (Khashabi et al., 2018), LGD (Linzen et al., 2016), 20 News (Lang, 1995),…”
ERASER: A Benchmark to Evaluate Rationalized NLP Models (2019, methodology): “…‘pipeline’ approaches in which independent models are trained to perform rationale extraction and classiﬁcation on the basis of these, respectively (Lehman et al., 2019; Chen et al., 2019 (Radford et al., 2018) on free-text rationales provided by humans, with an objective of generating…”
MSˆ2: Multi-Document Summarization of Medical Studies (2021, background): “We adapt the Evidence Inference (EI) dataset and models (DeYoung et al., 2020) for labeling.”
Evaluating and Characterizing Human Rationales (2020, unknown): “and Evidence Inference (Lehman et al., 2019).”
An Information Bottleneck Approach for Controlling Conciseness in Rationale Extraction (2020, methodology): “Evidence Inference (Lehman et al., 2019): A three-way classiﬁcation task over full-text scientiﬁc articles for inferring whether a given medical intervention is reported to either signiﬁcantly increase , signiﬁcantly decrease , or have no signiﬁcant effect on a speciﬁed outcome compared to a…”
Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, background): “0 [71] evidence inference none experts 10,137 EVIDENCEINFERENCE V1.”
Cross-Domain Transfer of Generative Explanations Using Text-to-Text Models (2021, methodology): “Recently, rationalization pipelines have been applied to medical and scientific text, for instance [42] and [24] utilize BERT-to-BERT models for SciFact and for classifying random clinical trials, respectively.”
Fact or Fiction: Verifying Scientific Claims (2020, background): “The evidence inference task (Lehman et al., 2019), involves predicting the effect of a medical intervention on a speciﬁed outcome.”
Learning to Faithfully Rationalize by Construction (2020, unknown): “In Lei et al. (2016)’s framework, direct supervision provides considerable performance improvement in the case of Evidence Inference (although still suffering from variance effects), and did not affect performance on MultiRC.”
Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, methodology): “0 (Lehman et al., 2019) evidence inference none experts 10,137”
Aligning Faithful Interpretations with their Social Attribution (2020, background): “(Lehman et al., 2019) Based on biomedical articles of controlled trials, and a given intervention, the task is to infer the outcome of the experiment as a result of the intervention (among significantly increased, significantly decreased, and no significant change), and give supporting portions of…”
Your fairness may vary: Pretrained language model fairness in toxic text classification (2021, background): “fields such as medical evidence inference (Lehman et al., 2019; Subramanian et al., 2020) and chemistry (Schwaller et al.”
On the Diversity and Limits of Human Explanations (2021, background): “Lehman et al. (2019) Question answering Prompt Generation: Question answering & Prompt creators were instructed to identify a snippet, in a given full-text article, that reports a relationship between an intervention, comparator, and outcome.”