ExpBERT: Representation Engineering with Natural Language Explanations

By Shikhar Murty, Pang Wei Koh, Percy Liang. 2020

45 citations recorded by the source corpus.

Suppose we want to specify the inductive bias that married couples typically go on honeymoons for the task of extracting pairs of spouses from text. In this paper, we allow model developers to specify these types of inductive biases as natural language explanations. We use BERT fine-tuned on MultiNLI to “interpret” these explanations with respect to the input sentence, producing explanation-guided representations of the input. Across three relation extraction tasks, our method, ExpBERT, matches a BERT baseline but with 3–20x less labeled data and improves on the baseline by 3–10 F1 points with the same amount of labeled data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Percy Liang
3. Jianfeng Gao
4. Yejin Choi
5. Luke S. Zettlemoyer
6. Quoc V. Le
7. Noah A. Smith
8. Yonatan Belinkov

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Ming-Hsuan Yang
5. Xiaogang Wang
6. Jitendra Malik
7. Andrew Zisserman
8. Andrea Vedaldi

In Large Language Models

1. Percy Liang
2. Yonatan Belinkov
3. Yejin Choi
4. Maosong Sun
5. Zhou Yu
6. Ilya Sutskever
7. Luke S. Zettlemoyer
8. Chenyan Xiong

Citations in the corpus

• Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “More recently, ExpBERT (Murty et al., 2020) uses feature representations produced by conditioning a deep contextual language model on natural language explanations and descriptions of relations to improve performance on the task of relation extraction.”
• ALICE: Active Learning with Contrastive Natural Language Explanations (2020, background): “Murty et al. (2020); He and Peng (2017) encode natural language explanations as additional features to assist classification.”
• Towards Interpretable Natural Language Understanding with Explanations as Latent Variables (2020, methodology): “Some early work [11, 9, 10] exploited NL explanation as additional features for prediction.”
• Concept Bottleneck Models (2020, background): “, 2003; Bunescu & Mooney, 2005), or natural language explanations (Murty et al., 2020).”
• Natural Language Inference with a Human Touch: Using Human Explanations to Guide Model Attention (2021, methodology): “, 2019), relation extraction (Murty et al., 2020), visual classification tasks (Liang et al.”
• Supervising Model Attention with Human Explanations for Robust Natural Language Inference (2021, background): “More widely, explanations have improved performance on a range of domains, including commonsense reasoning (Rajani et al. 2019), relation extraction (Murty, Koh, and Liang 2020) and visual classification tasks (Liang, Zou, and Yu 2020; Mu, Liang, and Goodman 2020).”