Sequence Transduction with Recurrent Neural Networks

By Alex Graves. 2012

2,118 citations recorded by the source corpus.

Many machine learning tasks can be expressed as the transformation---or \emph{transduction}---of input sequences into output sequences: speech recognition, machine translation, protein secondary structure prediction and text-to-speech to name but a few. One of the key challenges in sequence transduction is learning to represent both the input and output sequences in a way that is invariant to sequential distortions such as shrinking, stretching and translating. Recurrent neural networks (RNNs) are a powerful sequence learning architecture that has proven capable of learning such representations. However RNNs traditionally require a pre-defined alignment between the input and output sequences to perform transduction. This is a severe limitation since \emph{finding} the alignment is the most difficult aspect of many sequence transduction problems. Indeed, even determining the length of the output sequence is often challenging. This paper introduces an end-to-end, probabilistic sequence transduction system, based entirely on RNNs, that is in principle able to transform any input sequence into any finite, discrete output sequence. Experimental results for phoneme recognition are provided on the TIMIT speech corpus.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, methodology): “The Connectionist Sequence Classific ation is another popular technique for mapping sequences to sequences with neural networks, altho ugh it assumes a monotonic alignment between the inputs and the outputs [11, 9].”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “A similar approach of aligning an output symbol with an input symbol was proposed recently by Graves (2013) in the context of handwriting synthesis.”
Generating Sequences With Recurrent Neural Networks (2013, background): “One neural network model able to make sequential predictions based on two sequences of different length and unknown alignment is the RNN transducer [9].”
Speech recognition with deep recurrent neural networks (2013, methodology): “For a length U target sequence z, the complete set of TU decisions jointly determines a distribution over all possible alignments between x and z, which can then be integrated out with a forwardbackward algorithm to determine log Pr(z|x) [10].”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “introduced recently to improve the quality of LMgenerated text (Graves, 2012; Fan et al., 2018; Holtzman et al., 2020; Meister et al., 2022).”
Imputer: Sequence Modelling via Imputation and Dynamic Programming (2020, methodology): “, the linear bottleneck between the prediction and transcription network (Graves, 2012)); this has typically limited the empirical”
BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “As a baseline, we have listed SoTA word error rates for each task inside SpeechStew, and recorded the performance of the ConformerXXL RNN-T network pretrained with Libri-Light data trained on SpeechStew [30].”
Unsupervised Speech Recognition (2021, background): “Such models are often classified into two categories: hybrid (Juang et al., 1986; Young, 1996; Povey, 2005; Hinton et al., 2012; Abdel-Hamid et al., 2012; Bourlard and Morgan, 2012) or end-to-end (Graves et al., 2006; Graves, 2012; Chorowski et al., 2015; Rao et al., 2017).”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, positive): “In our implementation, SpeechStew uses the Conformer [29] RNN-T [30] architecture.”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “effectively. Index Terms— multitask, text-only, end-to-end, LibriSpeech 1. INTRODUCTION Attention-based end-to-end (E2E) speech recognition systems map audiofeaturesdirectlytoatext-levelrepresentation[1,2,3,4,5,6,7, 8, 9, 10, 11]. Such models are trained on transcribed speech datasets comprised of p”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “The ASR network is a sequence transducer [46] consisting of a LSTM [47] decoder and a Conformer [5] encoder, whose major component is a stack of "conformer block"s, each of which is a series of (2)We emphasize that making the ASR model bigger on its own does not result in performance gains—only upon”
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters (2020, background): “In [7] and [8], authors train a seq2seq and RNN-T [13] models respectively on 9 Indian languages.”
Improved Noisy Student Training for Automatic Speech Recognition (2020, methodology): “We introduce a coverage penalty term [31] in the fusion score with parameter c for LAS models [32], while we use a constant non-blank reward ρ to shift the logits produced by the LM for RNN-T [33] networks [34, 35].”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Recurrent neural networks (RNNs) have been the defacto choice for ASR [1, 2, 3, 4] as they can model the temporal dependencies in the audio sequences effectively [5].”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “Our network is based on the RNN-Transducer framework [18, 19, 20].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “We then use the forward algorithm [10, 17] to compute the last alpha variable α(T,U), which corresponds to P (y|x) defined in Eq.”
On the Properties of Neural Machine Translation: Encoder–Decoder Approaches (2014, methodology): “The beam-search to (approximately) find a sequence of maximum log-probability under RNN was proposed and used successfully in (Graves, 2012) and (Boulanger-Lewandowski et al.”
Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models (2015, methodology): “We also considered the use of beam-search for RNNs (Graves 2012) to approximate the most probable (MAP) last utterance, U 3 , given the ﬁrst two utterances, U 1 and U 2 .”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, background): “ately, each with a different objective. Recent work in this area attempts to rectify this disjoint training issue by designing models that are trained end-to-end – from speech directly to transcripts [10, 11, 12, 13, 14, 15]. Two main approaches for this are Connectionist Temporal Classiﬁcation (CTC”
Attention-Based Models for Speech Recognition (2015, background): “Second, the alignment mechanism of both the CTC and RNN Transducer is constrained to be “monotonic” to keep marginalization of the alignment tractable.”