FlowSeq: Non-Autoregressive Conditional Sequence Generation with Generative Flow

By Xuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, Eduard H. Hovy. 2019

203 citations recorded by the source corpus.

Most sequence-to-sequence (seq2seq) models are autoregressive; they generate each token by conditioning on previously generated tokens. In contrast, non-autoregressive seq2seq models generate all tokens in one pass, which leads to increased efficiency through parallel processing on hardware such as GPUs. However, directly modeling the joint distribution of all tokens simultaneously is challenging, and even with increasingly complex model structures accuracy lags significantly behind autoregressive models. In this paper, we propose a simple, efficient, and effective model for non-autoregressive sequence generation using latent variable models. Specifically, we turn to generative flow, an elegant technique to model complex distributions using neural networks, and design several layers of flow tailored for modeling the conditional density of sequential latent variables. We evaluate this model on three neural machine translation (NMT) benchmark datasets, achieving comparable performance with state-of-the-art non-autoregressive NMT models and almost constant decoding time w.r.t the sequence length.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech (2021, neutral): “Similar to our work, Ma et al. (2019) proposed a conditional VAE with normalizing ﬂows in a conditional prior network for non-autoregressive neural machine translation, FlowSeq.”
Understanding Knowledge Distillation in Non-autoregressive Machine Translation (2019, methodology): “In general, all the NAT models except FlowSeq and LevT-big adopts a similar architecture and hyper-parameters as the Transformer-base (see Tab.”
Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation (2020, methodology): “…al., 2019b; Shao et al., 2020; Tu et al., 2020; Saharia et al., 2020; Ghazvininejad et al., 2020a), reordering input sentences (Ran et al., 2021), training on additional data from an autoregressive model (Zhou & Keung, 2020), and modeling with latent variables (Ma et al., 2019; Shu et al., 2020).”
Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade (2020, methodology): “…advantage, fully NAT system – model makes parallel predictions with single neural network forward – still lags behind in translation quality and has not been fully explored in literature (Libovick´y and Helcl, 2018; Li et al., 2018; Sun et al., 2019; Ma et al., 2019; Ghazvininejad et al., 2020a).”
Universal Multimodal Representation for Language Understanding (2023, background): “As a result, the visual information is only applied to the translation task over speciﬁc multimodal datasets [25, 26, 27, 28, 29], instead of general text-only NMT [30, 31, 32] and low-resource text-only NMT [33, 34, 35, 36].”
Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “…reﬁnement (Lee et al., 2018), CTC models (Libovicky and Helcl, 2018), insertion-based methods (Stern et al., 2019; Chan et al., 2019b), edit-based methods (Gu et al., 2019; Ruis et al., 2019), masked language models (Ghazvininejad et al., 2019, 2020b), and normalizing ﬂow models (Ma et al., 2019).”
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, methodology): “Recent work proposed methods to mitigate this multimodality issue, including iterative reﬁnement (e.g., Lee et al., 2018; Ghazvininejad et al., 2019; Gu et al., 2019b; Kasai et al., 2020) and modeling with latent variables (e.g., Ma et al., 2019; Shu et al., 2020).”
Step-unrolled Denoising Autoencoders for Text Generation (2021, methodology): “FlowSeq (Ma et al., 2019) applied Normalising Flows to machine translation.”
Rule-based Morphological Inflection Improves Neural Terminology Translation (2021, methodology): “…Constraints Instead of generating the output sequence incrementally from left to right, non-autoregressive NMT generates tokens in parallel (Gu et al., 2018; van den Oord et al., 2018; Ma et al., 2019) or by iteratively editing an initial sequence (Lee et al., 2018; Ghazvininejad et al., 2019).”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “Many latent variable models for text use LSTMs (Hochreiter and Schmidhuber, 1997) as their decoders (Yang et al., 2017; Ziegler and Rush, 2019; Ma et al., 2019).”
Contextualized Perturbation for Textual Adversarial Attack (2020, background): “…lexical substitution (Wu et al., 2019a; Zhou et al., 2019a; Qiang et al., 2020; Wu et al., 2019b, inter alia ) to non-autoregressive generation (Gu et al., 2018; Lee et al., 2018; Ghazvininejad et al., 2019; Ma et al., 2019; Sun et al., 2019; Ren et al., 2020; Zhang et al., 2020b, inter alia ).”
Improving Non-autoregressive Neural Machine Translation with Monolingual Data (2020, methodology): “We report tokenized BLEU scores in line with prior work (Lee et al., 2018; Ma et al., 2019), which are case-insensitive for WMT16 En-Ro and case-sensitive for WMT14 En-De in the data provided by Lee et al.”
ENGINE: Energy-Based Inference Networks for Non-Autoregressive Machine Translation (2020, methodology): “, 2020) and rescoring with autoregressive models (Kaiser et al., 2018; Ma et al., 2019; Sun et al., 2019).”
Civil Rephrases Of Toxic Texts With Self-Supervised Transformers (2021, background): “In the future, we plan to explore whether the decoding can beneﬁt from NAR generation (Ma et al., 2019; Ren et al., 2020).”
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2020, background): “However, their output quality suffers due to the large decoding space and strong independence assumptions between target tokens (Ma et al., 2019; Wang et al., 2019).”