Unsupervised Neural Machine Translation with Weight Sharing

By Zhen-Le Yang, Feng Wang, Bo Xu. 2018

131 citations recorded by the source corpus.

Unsupervised neural machine translation (NMT) is a recently proposed approach for machine translation which aims to train the model without using any labeled data. The models proposed for unsupervised NMT often use only one shared encoder to map the pairs of sentences from different languages to a shared-latent space, which is weak in keeping the unique and internal characteristics of each language, such as the style, terminology, and sentence structure. To address this issue, we introduce an extension by utilizing two independent encoders but sharing some partial weights which are responsible for extracting high-level representations of the input sentences. Besides, two different generative adversarial networks (GANs), namely the local GAN and global GAN, are proposed to enhance the cross-language translation. With this new approach, we achieve significant improvements on English-German, English-French and Chinese-to-English translation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, neutral): “uage Generation Method Setting en - fr fr - en en - de de - en en - ro ro - en Artetxe et al. (2017) 2-layer RNN 15.13 15.56 6.89 10.16 - - Lample et al. (2017) 3-layer RNN 15.05 14.31 9.75 13.33 - - Yang et al. (2018) 4-layer Transformer 16.97 15.58 10.86 14.62 - - Lample et al. (2018) 4-layer Tran”
Meta-Learning for Low-Resource Neural Machine Translation (2018, background): “…such as multi-task learning (Gulcehre et al., 2015; Zhang and Zong, 2016), back-translation (Sen-nrich et al., 2015), dual learning (He et al., 2016) and unsupervised machine translation with mono-lingual corpora only for both sides (Artetxe et al., 2017b; Lample et al., 2017; Yang et al., 2018).”
Phrase-Based & Neural Unsupervised Machine Translation (2018, background): “Finally, our work can be seen as an extension of recent studies (Lample et al., 2018; Artetxe et al., 2018; Yang et al., 2018) on fully unsupervised MT with two major contributions.”
Unsupervised Statistical Machine Translation (2018, methodology): “(from WMT 2016), as this was used by some previous work in unsupervised NMT (Lample et al., 2018; Yang et al., 2018)5.”
Extract and Edit: An Alternative to Back-Translation for Unsupervised Neural Machine Translation (2019, neutral): “Many research efforts have been spent on how to utilize the monolingual data to improve the NMT system when only limited supervision is available (Gulcehre et al., 2015; Sennrich et al., 2016a; He et al., 2016; Zhang and Zong, 2016; Yang et al., 2018).”
Unsupervised Pivot Translation for Distant Languages (2019, background): “Yang et al. (2018) partially share the parameter of the encoder and de-coder to enhance the semantic alignment between source and target language.”
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation (2019, methodology): “By combining CMLM and MLM, we can enforce 2018) are mainly the modiﬁcations of the encoder-decoder structure.”
Unsupervised Parallel Sentence Extraction with Parallel Segment Detection Helps Machine Translation (2019, methodology): “Unsupervised BWEs are used to make initial word-by-word translating systems which are then improved by iterative back-translation (Sennrich et al., 2016) us-ing neural systems (Lample et al., 2018a; Artetxe et al., 2018c; Yang et al., 2018).”
Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “To overcome the need for labelled data, unsupervised neural machine translation (USNMT) (Lample et al., 2018a; Artetxe et al., 2018b; Yang et al., 2018) focuses on the exploitation of very large amounts of monolingual sentences by combining denoising autoencoders with back-translation and…”
Hate speech detection: Challenges and solutions (2019, background): “, assigning a vector to a word, such as word2vec [25], are common when applying deep learning methods in natural language processing and text mining [26, 27].”
Cross-Lingual Training for Automatic Question Generation (2019, background): “Weight sharing: Based on the assumption that sentences and questions in two languages are similar in some latent space, in order to get a shared language independent representation, we share the last few layers of the encoder and the first few layers of the decoder (Yang et al., 2018).”
Translating Translationese: A Two-Step Approach to Unsupervised Machine Translation (2019, methodology): “Lample et al. (2018a,c), Artetxe et al. (2018), and Yang et al. (2018) use iterative back-translation to train MT models in both directions simultaneously.”
Machine Translation With Weakly Paired Bilingual Documents (2018, result): “Our method is compared with several previous works (Lample et al., 2018; 2017; Yang et al., 2018) in Table 2.”
An Effective Approach to Unsupervised Machine Translation (2019, methodology): “This method was further improved by Yang et al. (2018), who use two language-speciﬁc encoders sharing only a subset of their parameters, and incorporate a local and a global generative adversarial network.”
The Missing Ingredient in Zero-Shot Neural Machine Translation (2019, background): “Similar to our work, Artetxe et al. (2018); Lample et al. (2017); Yang et al. (2018) explore applying adversarial losses on the encoder to ensure that the representations are language agnostic.”
Neural Machine Translation for Low-resource Languages: A Survey (2021, background): “[204] proposed an architecture with two diferent GANs: one local GAN to constrain source and target languages to have the same distribution to try and fool the local discriminator, and one global GAN to ine-tune the corresponding generator.”