Elastic weight consolidation for better bias inoculation

By James Thorne, Andreas Vlachos. 2021

14 citations recorded by the source corpus.

The biases present in training datasets have been shown to affect models for sentence pair classification tasks such as natural language inference (NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate them, a common issue is that of catastrophic forgetting of the original training dataset. In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases while being less susceptible to catastrophic forgetting. In our evaluation on fact verification and NLI stress tests, we show that fine-tuning with EWC dominates standard fine-tuning, yielding models with lower levels of forgetting on the original (biased) dataset for equivalent gains in accuracy on the fine-tuning (unbiased) dataset.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

1. Sebastian Riedel
2. Amrith Krishna
3. Andreas Vlachos
4. James Thorne
5. Iryna Gurevych
6. Max Glockner

Citations in the corpus

• ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “To mitigate the effect of catastrophic forgetting, we apply L2 regularisation for all the three models (Thorne and Vlachos, 2021a) which improves all models on the FEVER development set.”
• Evidence-based Verification for Real World Information Needs (2021, background): “While there are some signals that the model can exploit, predicting veracity without evidence remains a challenging task – especially in comparison to previous datasets where models attain higher accuracies without evidence (Thorne and Vlachos, 2021, Appendix 1).”