Incorporating Discrete Translation Lexicons into Neural Machine Translation

By Philip Arthur, Graham Neubig, Satoshi Nakamura. 2016

217 citations recorded by the source corpus.

Neural machine translation (NMT) often makes mistakes in translating low-frequency content words that are essential to understanding the meaning of the sentence. We propose a method to alleviate this problem by augmenting NMT systems with discrete translation lexicons that efficiently encode translations of these low-frequency words. We describe a method to calculate the lexicon probability of the next word in the translation candidate by using the attention vector of the NMT model to select which source word lexical probabilities the model should focus on. We test two methods to combine this probability with the standard NMT probability: (1) using it as a bias, and (2) linear interpolation. Experiments on two corpora show an improvement of 2.0-2.3 BLEU and 0.13-0.44 NIST score, and faster convergence time.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

ShapeWorld - A new test methodology for multimodal language understanding (2017, background): “Deep networks for language tasks may exhibit similarly odd behavior (Sproat and Jaitly, 2016; Arthur et al., 2016).”
An Analysis of Encoder Representations in Transformer-Based Machine Translation (2018, background): “While other approaches make use of external information to improve the internal representation of NMT models (Arthur et al., 2016; Niehues and Cho, 2017; Alkhouli and Ney, 2017), the Transformer seems to be able to encode a lot of structural information without explicitly incorporating any…”
Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation (2018, background): “One topic that has received attention in the literature is the tendency of NMT systems to do poorly with rare words, and in particular, named entities (e.g., Arthur et al. (2016)).”
Stronger Baselines for Trustable Results in Neural Machine Translation (2017, background): “Lexicon bias: Incorporate scores from a pretrained lexicon (fast align model learned on the same data) directly as additional weights when selecting output words (Arthur et al., 2016).”
Guiding Neural Machine Translation with Retrieved Translation Pieces (2018, methodology): “A number of methods have been proposed to ameliorate these problems, including methods that incorporate symbolic knowledge such as discrete translation lexicons (Arthur et al., 2016; He et al., 2016; Chatterjee et al., 2017) and phrase tables (Zhang et al.”
Domain Robustness in Neural Machine Translation (2019, background): “Several authors present anecdotal evidence for NMT systems occasionally falling into a hallucination mode where translations are grammatically correct, but unrelated to the source sentence (Arthur et al., 2016; Koehn and Knowles, 2017; Nguyen and Chiang, 2018).”
RNN Approaches to Text Normalization: A Challenge (2016, background): “Interestingly, a recent paper (Arthur et al., 2016) discusses entirely analogous errors in neural machine translation, and attributes them to the same cause.”
Latent Relation Language Models (2019, background): “For instance in sequence transduction tasks, there are mechanisms to copy from the source sequence (Gu et al., 2016) or use word or phrase dictionaries (Arthur et al., 2016; Tang et al., 2016) to improve modeling of low-frequency words.”
Retrieval-based Neural Source Code Summarization (2020, methodology): “The NMT-based methods generally prefer high-frequency words in the corpus and may have trouble with low-frequency words [3, 64].”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, background): “, 2016), incorporating lexicons during decoding (Arthur et al., 2016), domain adaptation (Hu et al.”
Pre-Translation for Neural Machine Translation (2016, methodology): “Also the integration of lexical probabilities into NMT was successfully investigated (Arthur et al., 2016).”
Neural Machine Translation: A Review (2019, methodology): “alignment weights of the standard NMT attention model (He et al., 2016c; Arthur et al., 2016; Zhang & Zong, 2016a; Neubig, 2016; Tang et al., 2016).”
Improving Lexical Choice in Neural Machine Translation (2017, methodology): “The method of Arthur et al. (2016) does improve over the baseline NMT on most language pairs, but not by as much and as consistently as our models, and often not as well as Moses.”
Code-Switching for Enhancing NMT with Pre-Specified Translation (2019, methodology): “Arthur et al. (2016) incorporate discrete lexicons by converting lexicon probabilities into predictive probabilities and linearly interpolating them with NMT probability distributions.”
Jointly Learning to Align and Translate with Transformer Models (2019, methodology): “Word alignments have also been used for external dictionary assisted translation (Chatterjee et al., 2017; Alkhouli et al., 2018; Arthur et al., 2016) to improve translation of low frequency words or to comply with certain terminology guidelines.”