Sequence discriminative distributed training of long short-term memory recurrent neural networks

By Hasim Sak, Oriol Vinyals, Georg Heigold, Andrew W. Senior, Erik McDermott, Rajat Monga, Mark Z. Mao. 2014

148 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Optimization & Training

Citations in the corpus

Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, background): “Recurrent neural networks, typically LSTMs, are just beginning to be deployed in state-of-the art recognizers [24, 25, 55] and work well together with convolutional layers for the feature extraction [52].”
Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “se phrase-based SMT [47] to generate a sentence. In Section 6 we show that phrase-based SMT can be replaced with LSTMs for video description as has been shown previously for language translation [9], [65]. 7.2 Contemporaneous and Subsequent Work Similar work in activity recognition and visual descri”
Recurrent Neural Network Regularization (2014, background): “Recent work has shown that LSTMs can achieve excellent performance on acoustic modeling (Sak et al., 2014), yet relatively small LSTMs (in terms of the number of their parameters) can easily overﬁt the training set.”
Listen, attend and spell: A neural network for large vocabulary conversational speech recognition (2015, methodology): “Two main approaches for this are Connectionist Temporal Classiﬁcation (CTC) [10] and sequence to sequence models with attention [16].”
Deep learning in neural networks: An overview (2014, background): “On spoken digit recognition, LSTM stacks outperformed HMMs, despite making fewer assumptions about the domain.”
Fast and accurate recurrent neural network acoustic models for speech recognition (2015, methodology): “A number of sequence-level discriminative training criteria incorporating the lexical and language model constraints used in speech decoding have been shown to improve the performance of DNN and RNN acoustic models bootstrapped with CE [19, 20, 12, 21, 4] or CTC training criteria [5].”
Sequence-to-Sequence Learning as Beam-Search Optimization (2016, methodology): “The necessity of pre-training in this instance is consistent with the findings of other authors who train non-local neural models (Kingsbury, 2009; Sak et al., 2014; Andor et al., 2016; Ranzato et al., 2016).5 Similarly, it is clear that the smaller the beam used in training is, the less room the…”
Exploring architectures, data and units for streaming end-to-end speech recognition with RNN-transducer (2017, methodology): “The acoustic model is a CTC trained LSTM that predicts context-dependent (CD) phonemes first fine-tuned with sequence discriminative training as described in [5] and further improved with word-level edit-based minimum Bayes risk (EMBR) proposed recently by Shannon [22].”
Parallel Multi-Dimensional LSTM, With Application to Fast Biomedical Volumetric Image Segmentation (2015, background): “neural networks (RNNs) initially designed for sequence processing. They achieved state-of-the-art results on challenging tasks such as handwriting recognition [3], large vocabulary speech recognition [4, 5] and machine translation [6]. Their architecture contains gates to store and read out informat”
A Framework for Parallel and Distributed Training of Neural Networks (2016, background): “It is also worth mentioning that a lot of research has been devoted recently to the design of parallel, asynchronous versions of stochastic gradient descent for training NNs on large clusters of commodity hardware (Dean et al., 2012; Sak et al., 2014; Abadi et al., 2016).”
Distilling Knowledge from Ensembles of Neural Networks for Speech Recognition (2016, methodology): “For this, we make a copy of the distilled student model and apply the sMBR training method of [19] on it.”
End-to-End Training of Acoustic Models for Large Vocabulary Continuous Speech Recognition with TensorFlow (2017, methodology): “Significant additional gains have resulted from switching from feed-forward DNNs to Recurrent Neural Networks (RNNs), in particular Long Short Term Memory Models (LSTMs), again trained both at the frame level and sequence level [4].”
On the compression of recurrent neural networks with an application to LVCSR acoustic modeling for embedded speech recognition (2016, methodology): “Systems are additionally adapted using the sMBR criterion [26, 27] on a set of”
Very Deep Convolutional Neural Networks for Noise Robust Speech Recognition (2016, methodology): “2) The LSTM-RNN acoustic model has achieved great success recently [34], [38], [39], with good performance on several tasks.”
Deep Learning for Audio Signal Processing (2019, background): “For example, for speech recognition, Mohamed et al. [10] showed that the activations at lower layers of DNNs can be thought of as speaker-adapted features, while the activations of the upper layers of DNNs can be thought of as performing class-based discrimination.”
Transferring knowledge from a RNN to a DNN (2015, methodology): “Finally, we did not experiment with sequence training [26], sequence training has almost always shown to help [27], it would be interesting to see the effects of sequence training on these small models, and whether we can further improve the ASR performance.”
Long-Term Recurrent Convolutional Networks for Visual Recognition and Description (2017, background): “In Section 6 we show that phrase-based SMT can be replaced with LSTMs for video description as has been shown previously for language translation [9], [65].”
Persistent RNNs: Stashing Recurrent Weights On-Chip (2016, methodology): “RNNs, typically LSTMs, arecommonly used in state-of-the art recognizers (Graves et al., 2013; H. Sak et al., 2014; Sak et al., 2014) and work well together with convolutional layers for feature extraction (Sainath et al., 2015).”
Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition (2014, background): “Subsequently, the sequence discriminative training of LSTM networks is investigated in [17], and a significant gain was obtained.”
Conditional-Computation-Based Recurrent Neural Networks for Computationally Efficient Acoustic Modelling (2018, background): “In recent years deep bidirectional RNN’s have set the state-ofthe-art performance in several ASR tasks [1, 2, 3, 4, 5, 6].”