A Neural Transducer

By Navdeep Jaitly, David Sussillo, Quoc V. Le, Oriol Vinyals, Ilya Sutskever, Samy Bengio. 2015

49 citations recorded by the source corpus.

Sequence-to-sequence models have achieved impressive results on various tasks. However, they are unsuitable for tasks that require incremental predictions to be made as more data arrives or tasks that have long input sequences and output sequences. This is because they generate an output sequence conditioned on an entire input sequence. In this paper, we present a Neural Transducer that can make incremental predictions as more input arrives, without redoing the entire computation. Unlike sequence-to-sequence models, the Neural Transducer computes the next-step distribution conditioned on the partially observed input sequence and the partially generated sequence. At each time step, the transducer can decide to emit zero to many output symbols. The data can be processed using an encoder and presented as input to the transducer. The discrete decision to emit a symbol at every time step makes it difficult to learn with conventional backpropagation. It is however possible to train the transducer by using a dynamic programming algorithm to generate target discrete decisions. Our experiments show that the Neural Transducer works well in settings where it is required to produce output predictions as data come in. We also find that the Neural Transducer performs well for long sequences even when attention mechanisms are not used.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

In Signal Processing

Citations in the corpus

Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “In the context of speech recognition, (Jaitly et al., 2015) proposed the Neural Transducer which segments sequences in non-overlapping chunks and attention is performed in each chunk independently.”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “Streamable end-to-end modeling architectures such as the Recurrent Neural Network Transducer (RNN-T) [10, 11, 12], Recurrent Neural Aligner (RNA) [13], and Neural Transducer [14] utilize an encoderdecoder based framework where both encoder and decoder are layers of RNNs that generate features from a”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “We achieve better performance than recently proposed sequence-to-sequence models (Luo et al., 2016; Jaitly et al., 2015), though the Table 1.”
The Neural Noisy Channel (2016, background): “Another trend of work that is related to our model is the investigation of making online prediction for machine translation (Gu et al., 2016; Grissom II et al., 2014; Sankaran et al., 2010) and speech recognition (Hwang & Sung, 2016; Jaitly et al., 2016).”
Monotonic Chunkwise Attention (2017, methodology): “A similar model to MoChA is the “Neural Transducer” (Jaitly et al., 2015), where the input sequence is pre-segmented into equally-sized non-overlapping chunks and attentive sequence-to-sequence transduction is performed over each chunk separately.”
Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, result): “2In the authors’ experience, RNN-T models are better suited for streaming long-form ASR tasks, compared to other E2E variants [20, 21, 22].”
Learning online alignments with continuous rewards policy gradient (2016, methodology): “[15] used an online sequence-to-sequence method with conditioning on partial inputs, which yielded encouraging results on the TIMIT dataset.”
Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning (2020, methodology): “The development of these streaming, end-to-end architectures has seen considerable attention for the tasks of automatic speech recognition [7, 8, 9, 10] and machine translation (MT) [11, 12, 13, 14].”
Recurrent Neural Aligner: An Encoder-Decoder Neural Network Model for Sequence to Sequence Mapping (2017, methodology): “Recurrent neural network models within encoder-decoder framework with attention mechanism have been commonly used for machine translation [4, 5] and speech recognition [6, 7, 8, 9, 10, 11].”
Recent Advances in Recurrent Neural Networks (2017, methodology): “Iterations of the CTC model, such as the sequence transducer and neural transducer [89], [103] have incorporated a second RNN to act as a language model to tackle tasks such as online speech recognition.”
Morphological Inflection Generation with Hard Monotonic Attention (2016, background): “Jaitly et al. (2015) proposed the Neural Transducer model, which is also trained on external alignments.”
Online Segment to Segment Neural Transduction (2016, background): “Jaitly et al. (2016) propose an online sequence to sequence model with attention that conditions on ﬁxed-sized blocks of the input sequence and emits output tokens corresponding to each block.”
Towards Online End-to-end Transformer Automatic Speech Recognition (2019, methodology): “A simple way to process the encoder online is blockwise computation, as in [17, 19, 23].”