Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation

By James Thorne, Andreas Vlachos. 2020

13 citations recorded by the source corpus.

The biases present in training datasets have been shown to be affecting models for a number of tasks such as natural language inference(NLI) and fact verification. While fine-tuning models on additional data has been used to mitigate such biases, a common issue is that of catastrophic forgetting of the original task. In this paper, we show that elastic weight consolidation (EWC) allows fine-tuning of models to mitigate biases for NLI and fact verification while being less susceptible to catastrophic forgetting. In our evaluation on fact verification systems, we show that fine-tuning with EWC Pareto dominates standard fine-tuning, yielding models lower levels of forgetting on the original task for equivalent gains in accuracy on the fine-tuned task. Additionally, we show that systems trained on NLI can be fine-tuned to improve their accuracy on stress test challenge tasks with minimal loss in accuracy on the MultiNLI dataset despite greater domain shift.

Subfields: Optimization & Training, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

  1. Wen-tau Yih
  2. Angela Fan
  3. Patrick Lewis
  4. Regina Barzilay
  5. Douwe Kiela
  6. James Thorne
  7. Adam Fisch
  8. Vladimir Karpukhin

In Safety, Robustness & Interpretability

  1. James Thorne
  2. Sebastian Riedel
  3. Ethan Perez
  4. Tal Schuster
  5. Regina Barzilay
  6. Adam Fisch
  7. Tim Rocktaschel
  8. Fabio Petroni

Citations in the corpus