Dual Transfer Learning for Neural Machine Translation with Marginal Distribution Regularization

By Yijun Wang, Yingce Xia, Li Zhao, Jiang Bian, Tao Qin, Guiquan Liu, Tie-Yan Liu. 2018

66 citations recorded by the source corpus.

Neural machine translation (NMT) heavily relies on parallel bilingual data for training. Since large-scale, high-quality parallel corpora are usually costly to collect, it is appealing to exploit monolingual corpora to improve NMT. Inspired by the law of total probability, which connects the probability of a given target-side monolingual sentence to the conditional probability of translating from a source sentence to the target one, we propose to explicitly exploit this connection to learn from and regularize the training of NMT models using monolingual data. The key technical challenge of this approach is that there are exponentially many source sentences for a target monolingual sentence while computing the sum of the conditional probability given each possible source sentence. We address this challenge by leveraging the dual translation model (target-to-source translation) to sample several mostly likely source-side sentences and avoid enumerating all possible candidate source sentences. That is, we transfer the knowledge contained in the dual model to boost the training of the primal model (source-to-target translation), and we call such an approach dual transfer learning. Experiment results on English-French and German-English tasks demonstrate that dual transfer learning achieves significant improvement over several strong baselines and obtains new state-of-the-art results.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Graham Neubig
5. Furu Wei
6. Luke S. Zettlemoyer
7. Jianfeng Gao
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Tao Qin
2. Tie-Yan Liu
3. Xu Tan
4. Luc Van Gool
5. Yingce Xia
6. Lijun Wu
7. Guillaume Lample
8. Dacheng Tao

In Natural Language Processing

1. Tao Qin
2. Tie-Yan Liu
3. Graham Neubig
4. Xu Tan
5. Guillaume Lample
6. Furu Wei
7. Alexis Conneau
8. Lijun Wu

Citations in the corpus

• Microsoft Research Asia’s Systems for WMT19 (2019, background): “MADL is an enhanced version of dual learning (He et al., 2016; Wang et al., 2018).”
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, neutral): “96 Dual Transfer Learning (Wang et al., 2018) 32.”
• Achieving Human Parity on Automatic Chinese to English News Translation (2018, background): “Ever since dual learning was proposed, it has been successfully applied to various real-world problems such as question answering [35], image classification [47], image segmentation [25], image to image translation [50, 52, 24], face attribute manipulation [31], and machine translation [19, 43, 23, ”
• Phrase-Based & Neural Unsupervised Machine Translation (2018, methodology): “…is typically provided as a small set of parallel sentences (Sennrich et al., 2015a; Gul-cehre et al., 2015; He et al., 2016; Gu et al., 2018; Wang et al., 2018); large sets of parallel sentences in related languages (Firat et al., 2016; Johnson et al., 2016; Chen et al., 2017; Zheng et al.,…”
• Sentence-wise Smooth Regularization for Sequence to Sequence Learning (2018, unknown): “SougoKnowing (Wang et al. 2017) 24.00 xmunmt (Tan et al. 2017) 23.40 Transformer+Smooth 25.05 De!En Actor-critic (Bahdanau et al. 2016) 28.53 CNN-a (Gehring et al. 2016) 30.04 Dual transfer learning (Wang et al. 2018) 32.35 Transformer+Smooth 33.47 Table 2: Comparison with previous works. encoder an”
• Multi-Agent Dual Learning (2019, background): “It is also worth studying how to apply the framework to other dual learning paradigms, including dual supervised learning (Xia et al., 2017b) and dual transfer learning (Wang et al., 2018).”
• Sentiment Classification by Leveraging the Shared Knowledge from a Sequence of Domains (2019, background): “Transfer learning or domain adaptation uses labeled training examples from the source domain to help learning in the target domain that has no or very few labeled training examples [1,12,18,22,40].”
• A Study of Reinforcement Learning for Neural Machine Translation (2018, result): “Our work is also related with the research works that leverage monolingual data for improving NMT models (Zhang and Zong, 2016; Sennrich et al., 2015a; Wang et al., 2018; Xia et al., 2016; Cheng et al., 2016).”
• Efficient Bidirectional Neural Machine Translation (2019, unknown): “16 Dual Transfer Learning (Wang et al., 2018) 32.”
• Exploiting Monolingual Data at Scale for Neural Machine Translation (2019, background): “…data is an important research topic for NMT and there are plenty of works studying this problem (Gulcehre et al., 2015, 2017; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Zhang et al., 2018; Cheng et al., 2016; Wang et al., 2018; Chinea-Rios et al., 2017; Wu et al., 2018).”
• Bi-Directional Neural Machine Translation with Synthetic Parallel Data (2018, background): “Cheng et al. (2016) optimize arg max θ P ( f (cid:48) | f ; θ ) as an autoencoder; Wang et al. (2018) minimize the difference between P ( f ) and P ( f (cid:48) | θ ) based on the law of total probability, while He et al. (2016) set the quality of both e and f (cid:48) as rewards for reinforcement…”
• Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “[191] transferred the knowledge learned in this dual translation task into the primary source-to-target translation task.”