A neural interlingua for multilingual machine translation

By Yichao Lu, Phillip Keung, Faisal Ladhak, Vikas Bhardwaj, Shaonan Zhang, Jason Sun. 2018

129 citations recorded by the source corpus.

We incorporate an explicit neural interlingua into a multilingual encoder-decoder neural machine translation (NMT) architecture. We demonstrate that our model learns a language-independent representation by performing direct zero-shot translation (without using pivot translation), and by using the source sentence embeddings to create an English Yelp review classifier that, through the mediation of the neural interlingua, can also classify French and German reviews. Furthermore, we show that, despite using a smaller number of parameters than a pairwise collection of bilingual NMT models, our approach produces comparable BLEU scores for each language pair in WMT15.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Orhan Firat
2. Felix Stahlberg
3. Graham Neubig
4. Tao Qin
5. Tie-Yan Liu
6. Ankur Bapna
7. Jiatao Gu
8. Xu Tan

In Computer Vision and Pattern Recognition

1. Orhan Firat
2. Felix Stahlberg
3. Ankur Bapna
4. Tao Qin
5. Tie-Yan Liu
6. Graham Neubig
7. Jiatao Gu
8. Kyunghyun Cho

In Natural Language Processing

1. Orhan Firat
2. Felix Stahlberg
3. Graham Neubig
4. Philipp Koehn
5. Ankur Bapna
6. Jiatao Gu
7. Tie-Yan Liu
8. Tao Qin

Citations in the corpus

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “This also indicates why methods that explicitly force languages to share the same representation space (Arivazhagan et al., 2019; Gu et al., 2019; Lu et al., 2018) may be key to improving zero-shot translation performance.”
• Multilingual Neural Machine Translation with Knowledge Distillation (2019, background): “Therefore, multilingual NMT (Johnson et al., 2017; Firat et al., 2016; Ha et al., 2016; Lu et al., 2018) is developed which handles multiple language pairs in one model, greatly reducing the offline training and online serving cost.”
• Multilingual Neural Machine Translation with Language Clustering (2019, background): “Similarly, Lu et al. (2018) propose the neural in-terlingua, which is an attentional LSTM encoder to link multiple encoders and decoders for different language pairs.”
• From Bilingual to Multilingual Neural Machine Translation by Incremental Training (2019, background): “Recent work (Lu et al., 2018) show how to perform many to many translations with independent encoders and decoders just by sharing additional language-specific layers that transform the language-specific representations into a shared one (without the need of a pivot language).”
• Neural Machine Translation: A Review (2019, background): “[518] reported gains in zero-shot settings by adding a boldly named “neural interlingual” component”
• One-to-Many Multilingual End-to-End Speech Translation (2019, background): “Multilinguality has been widely explored in neural MT [22, 23, 24, 25, 26, 27], where it is now commonly performed using the target forcing mechanism [28, 29], which enables translation to many languages ({one,many}-to-many) without changing the underlying NMT architecture.”
• Improving Zero-shot Translation with Language-Independent Constraints (2019, background): “Lu et al. (2018) proposed to explicitly define a recurrent layer with a fixed number of states as “Interlingua” which resembles our attention-pooling models.”
• Three Strategies to Improve One-to-Many Multilingual Translation (2018, background): “Firat et al. (2016a) and Lu et al. (2018) further propose to share a universal attention mechanism for many-to-many translations.”
• (Self-Attentive) Autoencoder-based Universal Language Representation for Machine Translation (2018, background): “ without attention is able to generate a common representation between languages. Authors use the inner product or cosine distance to evaluate the distance between sentence representations. Recently, Lu et al., (2018) train single encoders and decoders for each language generating interlingual embed”
• A Survey of Multilingual Neural Machine Translation (2019, background): “To overcome this, an attention bridge network generates a fixed number of contextual representations that are input to the attention network [94, 149].”
• Gender Bias in Multilingual Neural Machine Translation: The Architecture Matters (2020, background): “Architectures in this category may vary between sharing some layers (Firat et al., 2017; Lu et al., 2018) or no sharing at all (Escolano et al.”
• Multilingual Machine Translation: Closing the Gap between Shared and Language-specific Encoder-Decoders (2020, background): “Previous works, e.g. (Firat et al., 2016a; Lu et al., 2018), suffered from attention mismatch which was solved by sharing intermediate layers.”