Sequence to Sequence Mixture Model for Diverse Machine Translation
By Xuanli He, Gholamreza Haffari, Mohammad Norouzi. 2018
58 citations recorded by the source corpus.
Sequence to sequence (SEQ2SEQ) models lack diversity in their generated translations. This can be attributed to their limitations in capturing lexical and syntactic variations in parallel corpora, due to different styles, genres, topics, or ambiguity of human translation process. In this paper, we develop a novel sequence to sequence mixture (S2SMIX) model that improves both translation diversity and quality by adopting a committee of specialized translation models rather than a single translation model. Each mixture component selects its own training dataset via optimization of the marginal log-likelihood, which leads to a soft clustering of the parallel corpus. Experiments on four language pairs demonstrate the superiority of our mixture model compared to SEQ2SEQ model with the standard and diversity encouraged beam search. Our mixture model incurs negligible additional parameters and no extra computation in the decoding time.
Subfields: Artificial Intelligence, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Quoc V. Le
- Graham Neubig
- Hannaneh Hajishirzi
- Percy Liang
- Kyunghyun Cho
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Felix Stahlberg
- Jianfeng Gao
- Kyunghyun Cho
- Quoc V. Le
Citations in the corpus
- Unsupervised Data Augmentation for Consistency Training (2019, background): “There are also recent works on generating diverse translations [22, 56, 32] that might lead to further improvements when used as data augmentations.”
- Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, background): “Among these variants, the hMup objective is also known as multiple choice learning (MCL) for an ensemble of learners where the oracle loss is minimized (Guzman-Rivera et al., 2012), and He et al. (2018) has considered the sMup objective to train a sequence-to-sequence mixture model.”
- Teacher-Student Networks with Multiple Decoders for Solving Math Word Problem (2020, background): “Shen et al. [2019] and He et al. [2018] used multiple decoders with uniform mixing coefficient to improve diversity in machine translation.”
- Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, background): “He, Haffari, and Norouzi (2018) developed a sequence-to-sequence mixture model to adopt a committee of translation models.”
- Mixture Content Selection for Diverse Sequence Generation (2019, methodology): “He et al. (2018); Shen et al. (2019) introduce mixture of decoders with uniform mixing coefficient to improve diversity in machine translation.”
- Neural Machine Translation: A Review (2019, background): “[153] who combined models that selected their training data based on marginal likelihood, and the UCAM submission to WMT18 [126] that ensembled different NMT architectures with each other.”
- FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (2019, unknown): “Following He et al. (2018) and Shen et al.”