Mixture Models for Diverse Machine Translation: Tricks of the Trade

By Tianxiao Shen, Myle Ott, Michael Auli, Marc'Aurelio Ranzato. 2019

169 citations recorded by the source corpus.

Mixture models trained via EM are among the simplest, most widely used and well understood latent variable models in the machine learning literature. Surprisingly, these models have been hardly explored in text generation applications such as machine translation. In principle, they provide a latent variable to control generation and produce a diverse set of hypotheses. In practice, however, mixture models are prone to degeneracies---often only one component gets trained or the latent variable is simply ignored. We find that disabling dropout noise in responsibility computation is critical to successful training. In addition, the design choices of parameterization, prior distribution, hard versus soft EM and online versus offline assignment can dramatically affect model performance. We develop an evaluation protocol to assess both quality and diversity of generations against multiple references, and provide an extensive empirical study of several mixture model variants. Our analysis shows that certain types of mixture models are more robust and offer the best trade-off between translation quality and diversity compared to variational models and diverse decoding approaches.\footnote{Code to reproduce the results in this paper is available at \url{this https URL}}

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

Citations in the corpus

fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “Alternatively, in a mixture-of-experts model, a criterion may implement EM-style training and backpropagate only through the expert that produces the lowest loss (Shen et al., 2019).”
FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow (2019, methodology): “Following (Shen et al., 2019), we analyze the output diversity of FlowSeq.”
Data Diversification: A Simple Strategy For Neural Machine Translation (2019, background): “Our approach is also related but substantially different from the mixture of experts for diverse MT [20], iterative back-translation [11] and copied monolingual data for NMT [2]; see the Appendix for further details about these comparisons.”
multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning (2021, background): “, 2019), (2) varying the hidden representations or using multiple decoders (Dai et al., 2017; Jain et al., 2017; Shen et al., 2019).”
Taming Sparsely Activated Transformer with Stochastic Experts (2021, methodology): “We consider two MoE models proposed in Shen et al. (2019), referred to as MoE(dec) and MoE(tok), respectively, and three variants of the Switch Transformer proposed in Fedus et al. (2021).”
Few-shot Sequence Learning with Transformers (2020, background): “Further, they have been shown to use context tokens appended to an input to adapt their generations or to switch between different tasks (Lample et al., 2019; Shen et al., 2019; Zellers et al., 2019; Keskar et al., 2019).”
Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts (2022, methodology): “Independently param-eterizing each expert may exacerbate overfitting since the number of parameters increases linearly with the number of experts (Shen et al., 2019).”
Unsupervised Quality Estimation for Neural Machine Translation (2020, methodology): “First , we consider three system variants with differences in architecture or training: RNN-based NMT (Bahdanau et al., 2015; Luong et al., 2015), Mixture of Experts (MoE, He et al., 2018; Shen et al., 2019; Cho et al., 2019) and model ensemble (Garmash and Monz, 2016).”
Mixture Content Selection for Diverse Sequence Generation (2019, methodology): “Here we compare the effect of number of mixtures in our S ELECTOR and Mixture Decoder (Shen et al., 2019).”
The Vendi Score: A Diversity Evaluation Metric for Machine Learning (2022, methodology): “In text, variants of this metric include PAIRWISE-BLEU (Shen et al., 2019) and D-LEX-SIM (Fomicheva et al.”
Teacher-Student Networks with Multiple Decoders for Solving Math Word Problem (2020, background): “Shen et al. [2019] and He et al. [2018] used multiple decoders with uniform mixing coefﬁcient to improve diversity in machine translation.”
Better Document-Level Machine Translation with Bayes’ Rule (2019, background): “Pairwise-BLEU (Shen et al., 2019) is a metric of measuring the similarity of candidate translations.”