Identifying and Controlling Important Neurons in Neural Machine Translation
By Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, James R. Glass. 2018
198 citations recorded by the source corpus.
Neural machine translation (NMT) models learn representations containing substantial linguistic information. However, it is not clear if such information is fully distributed or if some of it can be attributed to individual neurons. We develop unsupervised methods for discovering important neurons in NMT models. Our methods rely on the intuition that different models learn similar properties, and do not require any costly external supervision. We show experimentally that translation quality depends on the discovered neurons, and find that many of them capture common linguistic phenomena. Finally, we show how to control NMT translations in predictable ways, by modifying activations of individual neurons.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yonatan Belinkov
- Mohit Bansal
- Graham Neubig
- Yoav Goldberg
- Luke S. Zettlemoyer
- Percy Liang
- Kyunghyun Cho
- Felix Stahlberg
In Computer Vision and Pattern Recognition
- Yonatan Belinkov
- Felix Stahlberg
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Mohit Bansal
- Xiaogang Wang
- Andrew Zisserman
In Natural Language Processing
- Yonatan Belinkov
- Felix Stahlberg
- Graham Neubig
- Yoav Goldberg
- James R. Glass
- Mohit Bansal
- Kyunghyun Cho
- Samuel R. Bowman
In Machine Translation
- Yonatan Belinkov
- Jesse Vig
- Nadir Durrani
- Hassan Sajjad
- Fahim Dalvi
- James R. Glass
- Ivan Titov
- Elena Voita
Citations in the corpus
- The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (2019, methodology): “Bau et al. (2019) used SVCCA as one of the methods used for identifying important individual neurons in NMT models.”
- Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned (2019, methodology): “Recently Bau et al. (2019) proposed a method for identifying important individual neurons in NMT models.”
- Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “11 Other studies that perform interventions to interpret NLP models without involving probing classifiers (e.g., Bau et al. 2019; Lakretz et al. 2019; Vig et al. 2020) are left out of the present scope.”
- Analysis Methods in Neural Language Processing: A Survey (2018, background): “A number of studies evaluated the effect of erasing or masking certain neural network components, such as word embedding dimensions, hidden units, or even full words (Li et al., 2016b; Feng et al., 2018; Khandelwal et al., 2018; Bau et al., 2018).”
- Investigating Catastrophic Forgetting During Continual Training for Neural Machine Translation (2020, background): “Considering that the parameters in the same row or column are connected to the same neurons in the former or latter layer, we argue that some neurons are more important for the model, which is consistent with the conclusion of Bau et al. (2019).”
- A Multiscale Visualization of Attention in the Transformer Model (2019, methodology): “Finally, we would like to enable users to manipulate the model, either by modifying attention (Lee et al., 2017; Liu et al., 2018; Strobelt et al., 2018) or editing individual neurons (Bau et al., 2019).”
- Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML (2019, methodology): “Following several recent works [20, 25, 18, 16, 21, 8, 3] we measure this by applying Canonical Correlation Analysis (CCA) to the latent representations of the network.”
- Information-Theoretic Probing with Minimum Description Length (2020, background): “…Tiede-mann, 2018; Voita et al., 2019b; Clark et al., 2019; Kovaleva et al., 2019) or individual neurons (Karpa-thy et al., 2015; Pham et al., 2016; Bau et al., 2019), (ii) looking at model predictions using manually defined templates, either evaluating sensitivity to specific grammatical errors…”
- Investigating Multilingual NMT Representations at Scale (2019, methodology): “, 2018), language models (Saphra and Lopez, 2018) and NMT (Bau et al., 2018).”
- Towards Understanding Neural Machine Translation with Word Importance (2019, background): “Most of the existing work focus on the former thread, which analyzes the linguistic information embeded in the learned representations (Shi et al., 2016; Belinkov et al., 2017; Yang et al., 2019) or the hidden units (Ding et al., 2017; Bau et al., 2019).”
- What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models (2018, unknown): “Some details are presented in (Bau et al. 2019).”
- Visualizing Attention in Transformer-Based Language models (2019, methodology): “Finally, we would like to enable users to intervene in the model, either by modifying attention (Stro-belt et al., 2018) or manipulating individual neurons (Bau et al., 2019).”
- Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth (2020, background): “…al., 2019; Hermann & Lampinen, 2020), and dynamics (Maheswaranathan et al., 2019) to revealing hidden linguistic structure in large language models (Bau et al., 2019; Kudugunta et al., 2019; Wu et al., 2019; 2020) to applications in neuroscience (Shi et al., 2019; Li et al., 2019; Merel et al.,…”
- BERT Learns (and Teaches) Chemistry (2020, methodology): “BERT builds upon the encoder structure of the encoder-decoder architecture of transformer, which uses multi-head self attention [2, 3].”
- One Size Does Not Fit All: Comparing NMT Representations of Different Granularities (2019, result): “Other studies carried a more fine-grained neuron-level analysis for NMT and LM (Dalvi et al., 2019; Bau et al., 2019; Lakretz et al., 2019).”
- Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021, background): “Inspired by the analysis work on NMT (Bau ar X iv :2 10 3.”
- Compositional Explanations of Neurons (2020, methodology): “Our work extends existing work on generating explanations for individual neurons in deep representations [4, 5, 10–12, 27] , in Complex explanations.”
- Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, background): “…(Liu et al., 2016), incorporating lexicons during decoding (Arthur et al., 2016), domain adaptation (Hu et al., 2019), unsupervised MT (Ren et al., 2020) and automatic evaluation or analysis of translation models (Bau et al., 2018; Stanovsky et al., 2019; Neubig et al., 2019; Wang et al., 2020).”