Imputer: Sequence Modelling via Imputation and Dynamic Programming

By William Chan, Chitwan Saharia, Geoffrey E. Hinton, Mohammad Norouzi, Navdeep Jaitly. 2020

130 citations recorded by the source corpus.

This paper presents the Imputer, a neural sequence model that generates output sequences iteratively via imputations. The Imputer is an iterative generative model, requiring only a constant number of generation steps independent of the number of input or output tokens. The Imputer can be trained to approximately marginalize over all possible alignments between the input and output sequences, and all possible generation orders. We present a tractable dynamic programming training algorithm, which yields a lower bound on the log marginal likelihood. When applied to end-to-end speech recognition, the Imputer outperforms prior non-autoregressive models and achieves competitive results to autoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1 WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.

Subfields: Artificial Intelligence, Natural Language Processing, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Probabilistic & Bayesian Methods

Citations in the corpus

End-to-End Object Detection with Transformers (2020, background): “rd to batch) lead to the development of parallel sequence generation, in the domains of audio [29], machine translation [12,10], word representation learning [8], and more recently speech recognition [6]. We also combine transformers and parallel decoding for their suitable trade-o between computati”
Rethinking Attention with Performers (2020, background): “Several solutions have been proposed to address this issue (Beltagy et al., 2020; Gulati et al., 2020; Chan et al., 2020; Child et al., 2019; Bello et al., 2019).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “, 2020), or hand-crafted (Chan et al., 2020) distributions have been used to enable learning the edit distribution.”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “End-to-end speech recognition models [1, 2, 3] have seen remarkable success in recent years [4].”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…very closely related to DDPMs, there has also been work on non-autoregressive modeling of text sequences that can be regarded as discrete-space DDPMs with a forward process that masks or remove tokens [Lee et al., 2018, Gu et al., 2019, Stern et al., 2019, Chan et al., 2020, Saharia et al., 2020].”
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, background): “Several solutions have been proposed to address this issue [2, 24, 4, 8, 1].”
Structured Denoising Diffusion Models in Discrete State-Spaces (2021, background): “Other works have considered non-autoregressive translation and speech transcription via insertion and deletion [16, 33], masking [13], and iteratively-refined sequence alignments [5, 34].”
Step-unrolled Denoising Autoencoders for Text Generation (2021, background): “Finally, a recent line of work called Imputer (Chan et al., 2020; Saharia et al., 2020) achieves strong results in machine translation by aligning target to source via dynamic programming.”
Non-Autoregressive Machine Translation with Latent Alignments (2020, methodology): “(6) where a′ ∼ β′(ã, a) captures all possible alignments a′ consistent with (ã, a) (Chan et al., 2020).”
Instances as Queries (2021, background): “This correspondence exists in all query based framework [47, 15, 37, 8, 7] regardless of the specific instantiations and applications.”
Continuous diffusion for categorical data (2022, background): “More recently, Imputer (Chan et al., 2020; Saharia et al., 2020) achieved good results by optimizing alignment between source and target.”