On Adversarial Removal of Hypothesis-only Bias in Natural Language Inference

By Yonatan Belinkov, Adam Poliak, Stuart M. Shieber, Benjamin Van Durme, Alexander M. Rush. 2019

74 citations recorded by the source corpus.

Popular Natural Language Inference (NLI) datasets have been shown to be tainted by hypothesis-only biases. Adversarial learning may help models ignore sensitive biases and spurious correlations in data. We evaluate whether adversarial learning can be used in NLI to encourage models to learn representations free of hypothesis-only biases. Our analyses indicate that the representations learned via adversarial learning may be less biased, with only small drops in NLI accuracy.

Subfields: Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

WinoGrande (2019, methodology): “Following a rigorous process, a question is deemed valid if (1) the majority of the three workers chooses the correct answer option, (2) they agree that the two answer options are unambiguous (one option is clearly more plausible than the other), and (3) the question cannot be answered simply by wor”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, background): “Belinkov et al. (2019) mitigate biases through adversarial training, penalizing the model for encoding a hypothesis-only bias. Finally, Schuster et al. (2019) re-weight the loss of instances during training guided by the mutual information between tokens and the instance labels.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “…(Lewis and Fan, 2019), carefully designed model architecture (Agrawal et al., 2018; Zhang et al., 2016), or adversarial removal of class-indicative features from model’s internal representations (Ramakrish-nan et al., 2018; Zhang et al., 2018a; Belinkov et al., 2019; Grand and Belinkov, 2019).”
Look at the First Sentence: Position Bias in Question Answering (2020, background): “For instance, sentence classiﬁcation models in natural language inference and argument reasoning comprehension suffer from word statistics bias (Poliak et al., 2018; Minervini and Riedel, 2018; Kang et al., 2018; Belinkov et al., 2019; Niven and Kao, 2019).”
There is Strength in Numbers: Avoiding the Hypothesis-Only Bias in Natural Language Inference via Ensemble Adversarial Training (2020, methodology): “There is some evidence that adversarial training produces more robust models: Belinkov et al. (2019a) found that models trained on the SNLI dataset using adversarial training generalise better to other datasets.”
Towards Robustifying NLI Models Against Lexical Dataset Biases (2020, methodology): “Belinkov et al. (2019a,b) showed that adversarial removal methods can be effective for the hypothesis-only NLI bias.”
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (2019, methodology): “For NLI, Belinkov et al. (2019) use adversarial learning to remove hypothesis-related bias in the sentence representations.”
GLUCOSE: GeneraLized and COntextualized Story Explanations (2020, background): “ently narrow and prone to hidden curation biases (Sharma et al., 2018; Belinkov et al., 2019).”
Natural Language Inference with a Human Touch: Using Human Explanations to Guide Model Attention (2021, methodology): “While research to date on creating more robust NLI models has mostly focused on preventing models learning from biases or artefacts in their datasets (Sanh et al., 2020; Belinkov et al., 2019a; Mahabadi et al., 2020; Utama et al., 2020; He et al., 2019), we take a different approach.”
An Adversarial Winograd Schema Challenge at Scale (2019, methodology): “Belinkov et al. (2019) propose an adversarial removal technique for NLI which en- courages models to learn representations that are free of hypothesis-only biases.”
Learning from others' mistakes: Avoiding dataset biases without modeling them (2020, background): “Several active areas of research tackle this challenge by adversarial training (Belinkov et al., 2019; Stacey et al., 2020), example forgetting (Yaghoobzadeh et al., 2019) and dynamic loss adjustment (Cad`ene et al., 2019).”