Using Target-side Monolingual Data for Neural Machine Translation through Multi-task Learning

By Tobias Domhan, Felix Hieber. 2017

95 citations recorded by the source corpus.

The performance of Neural Machine Translation (NMT) models relies heavily on the availability of sufficient amounts of parallel data, and an efficient and effective way of leveraging the vastly available amounts of monolingual data has yet to be found. We propose to modify the decoder in a neural sequence-to-sequence model to enable multi-task learning for two strongly related tasks: target-side language modeling and translation. The decoder predicts the next target word through two channels, a target-side language model on the lowest layer, and an attentional recurrent model which is conditioned on the source representation. This architecture allows joint training on both large amounts of monolingual and moderate amounts of bilingual data to improve NMT performance. Initial results in the news domain for three language pairs show moderate but consistent improvements over a baseline trained on bilingual data only.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Tagged Back-Translation (2019, background): “…et al., 2015, 2017), back-translation (Sennrich et al., 2016; Poncelas et al., 2018), unsupervised machine translation (Lample et al., 2018a; Artetxe et al., 2018), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017).”
Text Repair Model for Neural Machine Translation (2019, background): “…al., 2007) and neural MT (Gülçehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual learning (Cheng et al., 2016; Di He and Ma, 2016; Yingce Xia and Liu, 2017), and multi-task learning (Domhan and Hieber, 2017).”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “Domhan and Hieber [41] followed a similar approach; however, they exploited the target side monolingual data and shared the decoder across the two tasks: the translation task and the target-side LM learning task.”
Enhancement of Encoder and Attention Using Target Monolingual Corpora in Neural Machine Translation (2018, methodology): “Similarly, there is a method of training language models, jointly with the translator, using multi-task learning (Domhan and Hieber, 2017).”
APE at Scale and Its Implications on MT Evaluation Biases (2019, background): “…(Brants et al., 2007) and neural MT (G¨ulc¸ehre et al., 2015, 2017), back-translation (Sennrich et al., 2016b), unsupervised NMT (Lample et al., 2018a; Artetxe et al., 2018a), dual learning (Cheng et al., 2016; He et al., 2016; Xia et al., 2017), and multi-task learning (Domhan and Hieber, 2017).”
Improving Zero-shot Translation with Language-Independent Constraints (2019, unknown): “While this resembles an auto-encoder which can be combined with translation (He et al., 2016; Domhan and Hieber, 2017), it is not necessary to minimize the auto-encoder likelihood as in the multitask approach (Niehues and Cho, 2017), but only the decoder-level similarity between the true target…”
Investigating Backtranslation in Neural Machine Translation (2018, background): “In such cases, it has become the norm to resort to back-translating freely available monolingual data [Sennrich et al., 2016a; Belinkov and Bisk, 2017; Domhan and Hieber, 2017] to create an additional synthetic parallel corpus [Sennrich et al., 2016a] for training an NMT model.”
Dictionary-based Data Augmentation for Cross-Domain Neural Machine Translation (2020, background): “…came into shape (Chu and Wang, 2018) focusing on 1) leveraging monolingual IND data (Zhang and Zong, 2016b; Cheng et al., 2016; Currey et al., 2017; Domhan and Hieber, 2017); 2) sentence selection from OOD parallel corpora (Wang et al., 2017; van der Wees et al., 2017); 3) synthesizing IND…”
Transfer Learning of Language-independent End-to-end ASR with Language Model Fusion (2018, methodology): “Apart from the external LM, the MTL approach with LM objective are investigated in [19, 23].”
Simple Fusion: Return of the Language Model (2018, background): “Alternatively, multi-task learning has been used to incorporate source-side (Zhang and Zong, 2016) and target-side (Domhan and Hieber, 2017) monolingual data.”
Adaptive Knowledge Sharing in Multi-Task Learning: Improving Low-Resource Neural Machine Translation (2018, background): “Various recent works have attempted to improve NMT by scaffolding translation task on a single auxiliary task (Domhan and Hieber, 2017; Zhang and Zong, 2016; Dalvi et al., 2017).”
A Survey of Domain Adaptation for Neural Machine Translation (2018, methodology): “The data used can be either in-domain monolingual corpora (Zhang and Zong, 2016b; Cheng et al., 2016; Currey et al., 2017; Domhan and Hieber, 2017), synthetic corpora (Sennrich et al.”
Improving Domain Adaptation Translation with Domain Invariant and Specific Information (2019, background): “Domhan and Hieber (2017) further extend this work by training the RNNLM part and translation part jointly.”
Towards Enhancing Faithfulness for Neural Machine Translation (2020, background): “Zhang and Zong (2016) and Domhan and Hieber (2017) proposed to use multi-task learning for incorporating source/target side monolingual data in NMT. Zhou et al. (2019) introduced noisy data with multi-task learning to improve the robustness of NMT.”
Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models (2020, background): “Similar to deep fusion, Domhan and Hieber (2017) feed the unnormalized output of LM to the decoder of NMT. Domhan and Hieber (2017) jointly train the LM and NMT using monolingual target-side data and parallel data, respectively.”
Tricks for Training Sparse Translation Models (2021, background): “In this work, we study multilingual machine translation as a multi-task learning problem ( Dong et al., 2015; Domhan and Hieber, 2017), where a single model is trained to translate between many language pairs ( Fan et al., 2021).”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Domhan et al. [161] proposed joint training on a large number of unpaired languages and a small number of language pairs to improve neural machine translation (NMT) performance.”
Neural Machine Translation: A Review (2019, background): “Alternatively, multi-task learning has been used to incorporate source-side [291] and target-side [292] monolingual data.”