Improving Lexical Choice in Neural Machine Translation

By Toan Q. Nguyen, David Chiang. 2017

88 citations recorded by the source corpus.

We explore two solutions to the problem of mistranslating rare words in neural machine translation. First, we argue that the standard output layer, which computes the inner product of a vector representing the context with all possible output word embeddings, rewards frequent words disproportionately, and we propose to fix the norms of both vectors to a constant value. Second, we integrate a simple lexical module which is jointly trained with the rest of the model. We evaluate our approaches on eight language pairs with data sizes ranging from 100k to 8M words, and achieve improvements of up to +4.3 BLEU, surpassing phrase-based translation in nearly all settings.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Root Mean Square Layer Normalization (2019, methodology): “replacing RMS with L2-Norm), which 196 has been observed to improve lexical selection [16].”
Masked Language Model Scoring (2019, background): “One can view these as exacerbations of the rare word problem due to overconﬁdent logits (Nguyen and Chiang, 2018), and of over-translation (Tu et al., 2016).”
Continual Learning for Neural Machine Translation (2021, methodology): “Based on the above observation, we propose to add a weight normalization module similar to Nguyen and Chiang (2018) in the linear projection.”
Domain Robustness in Neural Machine Translation (2019, background): “Several authors present anecdotal evidence for NMT systems occasionally falling into a hallucination mode where translations are grammatically correct, but unrelated to the source sentence (Arthur et al., 2016; Koehn and Knowles, 2017; Nguyen and Chiang, 2018).”
Revisiting Low-Resource Neural Machine Translation: A Case Study (2019, methodology): “Low-resource NMT without auxiliary data has received less attention; work in this direction includes (Östling and Tiedemann, 2017; Nguyen and Chiang, 2018).”
Self-Attentional Acoustic Models (2018, methodology): “We apply label smoothing [21] and ﬁx the target embedding norm to 1 [22].”
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2020, background): “For En-De and Ro-En, the source and target embeddings are tied with the output layer weights (Press and Wolf, 2017; Nguyen and Chiang, 2018).”
Bi-Directional Neural Machine Translation with Synthetic Parallel Data (2018, background): “The effectiveness of tying input/output target embeddings has been veriﬁed on several low-resource language pairs (Nguyen and Chiang, 2018).”
Saliency-driven Word Alignment Interpretation for Neural Machine Translation (2019, methodology): “Because of this, some studies, such as Nguyen and Chiang (2018); Zenkel et al. (2019) proposed either to add extra modules to generate higher quality word alignments, or to use these modules to further improve the model performance or inter-pretability.”
Neural Machine Translation: A Review (2019, background): “In contrast, traditional (symbolic) MT systems can easily use very large vocabularies (Heafield et al., 2013; Lin & Dyer, 2010; Chiang, 2007; Koehn, 2010).”