Lessons from Natural Language Inference in the Clinical Domain

By Alexey Romanov, Chaitanya P. Shivade. 2018

329 citations recorded by the source corpus.

State of the art models using deep neural networks have become very good in learning an accurate mapping from inputs to outputs. However, they still lack generalization capabilities in conditions that differ from the ones encountered during training. This is even more challenging in specialized, and knowledge intensive domains, where training data is limited. To address this gap, we introduce MedNLI - a dataset annotated by doctors, performing a natural language inference task (NLI), grounded in the medical history of patients. We present strategies to: 1) leverage transfer learning using datasets from the open domain, (e.g. SNLI) and 2) incorporate domain knowledge from external data and lexical sources (e.g. medical terminologies). Our results demonstrate performance gains using both strategies.

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyong Lu
2. Iz Beltagy
3. Yifan Peng
4. Lucy Lu Wang
5. Di Jin
6. Madeleine van Zuylen
7. Jay DeYoung
8. Yuhao Zhang

In Molecular Biology

1. Yifan Peng
2. Iz Beltagy
3. Lucy Lu Wang
4. Zhiyong Lu
5. Junichi Tsujii
6. Dan Jurafsky
7. Di Jin
8. Curtis P. Langlotz

Citations in the corpus

• Revealing the Dark Secrets of BERT (2019, neutral): “These results are consistent with similar studies, e.g., Yosinski et al. (2014)’s results on fine-tuning a convolutional neural network pre-trained on ImageNet or Romanov and Shivade (2018)’s results on transfer learning for medical natural language inference.”
• Publicly Available Clinical BERT Embeddings (2019, methodology): “On three of the five tasks (MedNLI, i2b2 2010, and i2b2 2012), clinically fine-tuned BioBERT shows improvements over BioBERT or general BERT.”
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters (2020, methodology): “Natural Language Inference We also evaluate on the clinical natural language inference task MEDNLI (Romanov and Shivade, 2018) that aims to classify sentence pairs into three categories: CONTRA - DICTION , ENTAILMENT , and NEUTRAL .”
• ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model (2023, background): “scales due to architecture design and hardware support, although some efficient algorithms are proposed for the medical domain.(20,33,34) One of the most important LLMs is T5, with 11 billion parameters proposed by Google.”
• MSˆ2: Multi-Document Summarization of Medical Studies (2021, background): “…et al., 2018; Li et al., 2016), relation extraction (Jain et al., 2020; Luan et al., 2018; Kringelum et al., 2016), QA (Abacha et al., 2019), NLI (Romanov and Shivade, 2018; Khot et al., 2018), summarization (Cachola et al., 2020; Chan-drasekaran et al., 2019), claim verification (Wadden et al.,…”
• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets (2019, methodology): “P: PubMed, P+M: PubMed + MIMIC-III For named entity recognition, we used a Bi-LSTM-CRF implementation as a sequence tag-ger (Huang et al., 2015; Si et al., 2019; Lample et al., 2016).”
• Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation (2020, methodology): “MedNLI labels are equally distributed over three labels: entailment , neutral , and contradiction 4 .”
• A generalist vision–language foundation model for diverse biomedical tasks (2023, unknown): “…medical narratives that hold potential for addressing real-world clinical needs: (1) medical natural-language inference, using the MedNLI dataset 42 , which tests the model’s comprehension in deducing hypotheses from provided premises; (2) treatment suggestions for radiation therapy and…”