Freezing Subnetworks to Analyze Domain Adaptation in Neural Machine Translation

By Brian Thompson, Huda Khayrallah, Antonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Rebecca Marvin, Paul McNamee, Jeremy Gwinnup, Tim Anderson, Philipp Koehn. 2018

56 citations recorded by the source corpus.

To better understand the effectiveness of continued training, we analyze the major components of a neural machine translation system (the encoder, decoder, and each embedding space) and consider each component’s contribution to, and capacity for, domain adaptation. We find that freezing any single component during continued training has minimal impact on performance, and that performance is surprisingly good when a single component is adapted while holding the rest of the model fixed. We also find that continued training does not move the model very far from the out-of-domain model, compared to a sensitivity analysis metric, suggesting that the out-of-domain model can provide a good generic initialization for the new domain.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Felix Stahlberg
2. Graham Neubig
3. Bill Byrne
4. Orhan Firat
5. Yang Feng
6. Philipp Koehn
7. Xipeng Qiu
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Felix Stahlberg
2. Yang Feng
3. Shuhao Gu
4. Philipp Koehn
5. Bill Byrne
6. Xian Li
7. Danielle Saunders
8. Orhan Firat

In Natural Language Processing

1. Felix Stahlberg
2. Graham Neubig
3. Orhan Firat
4. Bill Byrne
5. Xian Li
6. Philipp Koehn
7. Xipeng Qiu
8. Yang Feng

Citations in the corpus

• Simple, Scalable Adaptation for Neural Machine Translation (2019, background): “Thompson et al. (2018) fine-tune selected components of the base model architecture, in order to determine how much fine-tuning each component contributes to the final adaptation performance.”
• Tagged Back-Translation (2019, background): “…to data selection, as in van der Wees et al. (2017); Axelrod et al. (2011), or curriculum learning approaches like fine-tuning on in-domain data (Thompson et al., 2018; Hassan Sajjad and Nadir Durrani and Fahim Dalvi and Yonatan Belinkov and Stephan Vogel, 2017; Freitag and Al-Onaizan, 2016).”
• Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “In this sense, the work of Thompson et al. (2018) is most related to ours, which tries to understand the effectiveness of continued training for improving the in-domain performance.”
• Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation (2019, methodology): “WIPO De$En data are large enough to train strong indomain systems (Thompson et al., 2018), so we truncate to 100k lines to simulate a more interesting domain adaptation scenario.”
• Domain Adaptive Inference for Neural Machine Translation (2019, background): “Vilar (2018) regularizes parameters with an importance network, while Thompson et al. (2018) freeze subsets of the model parameters before finetuning.”
• Recipes for Adapting Pre-trained Monolingual and Multilingual Models to Machine Translation (2020, background): “Approaches based on freezing various model components have also been proposed (Thompson et al., 2018; Zoph et al., 2016), but have focused on RNN models pre-trained with parallel data, not transformer models pre-trained on monolingual data.”
• Finding Sparse Structures for Domain Specific Neural Machine Translation (2021, methodology): “• Layer Freeze (Thompson et al. 2018): We froze all model layers except for the top layers of both the encoder and decoder, then fine-tuned the top layers on the target domain data.”
• Neural Machine Translation: A Review (2019, background): “by freezing sub-networks (Thompson et al., 2018) or by only learning additional scaling factors for hidden units rather than full weights (Swietojanski & Renals, 2014; Vilar, 2018).”