On integrating a language model into neural machine translation

By Caglar Gulcehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Yoshua Bengio. 2017

119 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Tagged Back-Translation (2019, methodology): “Various meth-ods for using target-side LMs have proven effective for NMT (He et al., 2016; G¨ulc¸ehre et al., 2017), but have tended to be less successful than back-translation – for example, G¨ulc¸ehre et al. (2017) report under +0.5 B LEU over their base-line on EnDe newstest14, whereas Edunov et…”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, methodology): “…starting with semisupervised methods relying on back-translation (Sennrich et al., 2015), integration of a language model into the decoder (Gulcehre et al., 2017; Stahlberg et al., 2018) all the way to fully unsupervised approaches (Lample et al., 2018b; Artetxe et al., 2018), which use…”
An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model (2017, methodology): “Further, they were all limited to grapheme systems, with [7] and [9] focused on machine translation.”
Two New Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English (2019, methodology): “…starting with semisupervised methods relying on back-translation (Sennrich et al., 2015), integration of a language model into the decoder (Gulcehre et al., 2017; Stahlberg et al., 2018) all the way to fully unsupervised approaches (Lample et al., 2018b; Artetxe et al., 2018), which use…”
Text Repair Model for Neural Machine Translation (2019, methodology): “There are several approaches to leverage monolingual data for NMT: Language model fusion for both phrase-based (Brants et al., 2007) and neural MT (Gülçehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual…”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, methodology): “• computer vision: graph-based methods [25, 26], generative modelling [27, 28], consistency-based methods [29, 30, 31, 32], and contrastive methods [33, 34, 35, 36]; • machine translation (MT): integration of a language model trained on monolingual data [37, 38, 39], backtranslation [40, 41, 42], sy”
Neural Machine Translation: A Review (2019, methodology): “[271, 272] suggested to integrate a separately trained RNN-LM into the NMT decoder.”
Leveraging Large Language Models for Mental Health Prediction via Online Text Data (2023, background): “…instruction-finetuned LLMs, such as GPT-4 [12], PaLM [15], FLAN-T5 [16], LLaMA [82], Alpaca [81], contain hundreds of billions of parameters and achieves a promising performance on a variety of tasks, such as question answering [60, 70], logic reasoning [87, 95], machine translation [10, 34] etc. .”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, methodology): “Gulcehre et al. (2017) describe using language model (LM) to improve the performance of NMT using shallow and deep fusion.”
Why Catalan-Spanish Neural Machine Translation? Analysis, comparison and combination with standard Rule and Phrase-based technologies (2017, methodology): “…ones mentioned in previous: Byte Pair Encoding (BPE) (Sennrich et al., 2016), character-based (Costa-juss`a and Fonollosa, 2016), coverage (Tu et al., 2016), language model (Gulcehre et al., 2017), multilingual (Firat et al., 2017) and other strategies (Wu et al., 2016) affect this language pair.”
Confident Adaptive Language Modeling (2022, background): “The large size of LLMs, however, results in massive computational load that might be limiting for certain real-world applications (e.g., machine translation) [9; 30; 42; 49; 59; 63; 71].”
AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “…have long been considered a core problem in natural language processing and demonstrated remarkable abilities for tasks such as machine translation (Gulcehre et al., 2017; Baziotis et al., 2020), open-ended dialogue modeling (Hosseini-Asl et al., 2020; Thoppilan et al., 2022), and even code…”
Adapting Neural Machine Translation with Parallel Synthetic Data (2017, background): “Different works have tack-led the inclusion of monolingual data, either in source (Zhang and Zong, 2016b) and target language (Gulcehre et al., 2015, 2017).”
Exploiting Monolingual Data at Scale for Neural Machine Translation (2019, background): “How to effectively leverage monolingual data is an important research topic for NMT and there are plenty of works studying this problem (Gulcehre et al., 2015, 2017; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Zhang et al., 2018; Cheng et al., 2016; Wang et al., 2018; Chinea-Rios…”
Back-Translation Sampling by Targeting Difficult Words in Neural Machine Translation (2018, methodology): “Gulcehre et al. (2017) proposed two methods, shallow and deep fusion, for integrating a neural language model into the NMT sys-tem.”
Simple Fusion: Return of the Language Model (2018, methodology): “An early attempt to use LMs for NMT, also known as shallow fusion , combines LM and NMT scores at inference time in a log-linear model (Gulcehre et al., 2015, 2017).”