ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation

By Lifu Tu, Richard Yuanzhe Pang, Sam Wiseman, Kevin Gimpel. 2020

58 citations recorded by the source corpus.

We propose to train a non-autoregressive machine translation model to minimize the energy defined by a pretrained autoregressive model. In particular, we view our non-autoregressive translation system as an inference network (Tu and Gimpel, 2018) trained to minimize the autoregressive teacher energy. This contrasts with the popular approach of training a non-autoregressive model on a distilled corpus consisting of the beam-searched outputs of such a teacher model. Our approach, which we call ENGINE (ENerGy-based Inference NEtworks), achieves state-of-the-art non-autoregressive results on the IWSLT 2014 DE-EN and WMT 2016 RO-EN datasets, approaching the performance of autoregressive models.

Subfields: Artificial Intelligence, Model Architectures, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah A. Smith
2. Mohit Bansal
3. Graham Neubig
4. Tomasz Korbak
5. Germán Kruszewski
6. Yejin Choi
7. Jianfeng Gao
8. Luke S. Zettlemoyer

In Model Architectures

1. Noah A. Smith
2. Chris Dyer
3. Jungo Kasai
4. Hady ElSahar
5. Germán Kruszewski
6. Tomasz Korbak
7. James Cross
8. Hao Peng

In Machine Translation

1. Taylor Berg-Kirkpatrick
2. Noah A. Smith
3. Chris Dyer
4. Hady ElSahar
5. Marc Dymetman
6. Ming Zhou

Citations in the corpus

• A Distributional Approach to Controlled Text Generation (2020, methodology): “Tu et al. (2020) propose an energy-based method to perform inference networks from pretrained Non-Autoregressive Machine Translation models.”
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis-Hastings (2021, methodology): “…sequence models and generate sequences from the MLMs in a non(pseudo)-probabilistic manner (Wang and Cho, 2019; Ghazvininejad et al., 2019; Tu et al., 2020; Gu et al., 2017; Mansimov et al., 2019), our technique samples correctly from the energy networks underlying MLMs that correspond to…”
• Energy-Based Models for Code Generation under Compilability Constraints (2021, background): “Tu et al. (2020) proposed an energy-based inference networks for non-autoregressive machine translation.”
• Energy-Based Models for Continual Learning (2020, background): “, 2019; Tu & Gimpel, 2019), machine translation (Tu et al., 2020), text generation (Deng et al.”
• A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models (2022, methodology): “After that, there is a surge of methods based on deep generative models, such as Variational Autoencoders (VAEs) [48, 125, 138, 142, 149, 154], Generative Adversarial Networks (GANs) [117, 140], and Energy-based Models [8, 25, 135, 166].”
• Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, background): “…(Li et al., 2019), training with different objectives than vanilla cross-entropy (Libovick´y and Helcl, 2018; Wang et al., 2019b; Shao et al., 2020; Tu et al., 2020; Saharia et al., 2020; Ghazvininejad et al., 2020a), reordering input sentences (Ran et al., 2019), training on additional data from…”
• Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021, unknown): “EBM has also been recently used in non-autoregressive machine translation (Tu et al., 2020).”