Scalable Minimum Bayes Risk Training of Deep Neural Network Acoustic Models Using Distributed Hessian-free Optimization

By Brian Kingsbury, Tara N. Sainath, Hagen Soltau. 2012

250 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Deep Learning: Methods and Applications (2014, methodology): “Other learning algorithms such as Hessian free [195, 238] or Krylov subspace methods [378] have shown a similar ability.”
Fast and accurate recurrent neural network acoustic models for speech recognition (2015, methodology): “A number of sequence-level discriminative training criteria incorporating the lexical and language model constraints used in speech decoding have been shown to improve the performance of DNN and RNN acoustic models bootstrapped with CE [19, 20, 12, 21, 4] or CTC training criteria [5].”
Training RNNs as Fast as CNNs (2017, methodology): “The ASR performance can be further improved by using bidirectional model and state-level Minimum Bayes Risk (sMBR) training (Kingsbury et al., 2012).”
A scalable approach to using DNN-derived features in GMM-HMM based acoustic modeling for LVCSR (2013, methodology): “The main factors contributed to the improved recognition accuracy compared with the traditional GMM-HMM based approach include the use of long-span features in the input layer of a DNN, a hierarchical highly nonlinear feature mapping due to its deep structure, and using decision-tree based HMM…”
A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property (2013, background): “…deep neural networks (DNN) that drastically cut errors in large vocabulary speech recognition (Yu et al., 2010; Dahl et al., 2011; Seide et al., 2011; Dahl et al., 2012; Kingsbury et al., 2012; Hinton et al., 2012; Deng et al., 2013b,c; Yu et al., 2013; Dahl et al., 2013; Sainath et al., 2013).”
Investigation of unsupervised adaptation of DNN acoustic models with filter bank input (2014, unknown): “, sequence training [16]).”
Differentiable Weighted Finite-State Transducers (2020, methodology): “In some cases, WFSTs are used statically to incorporate fixed lattices in discriminative sequence criteria (Kingsbury, 2009; Kingsbury et al., 2012; Su et al., 2013; Veselỳ et al., 2013).”
Rectifier Nonlinearities Improve Neural Network Acoustic Models (2013, background): “tions during DNN training, and claim state-of-the-art results on many challenging tasks in speech recognition (Hinton et al., 2012; Kingsbury et al., 2012; Vesely et al., 2013).”
Knowledge distillation for small-footprint highway networks (2016, background): “While the previous two loss functions are defined at the frame level, sequence training defines the loss at the sequence level, which usually yields significant improvement for speech recognition [16, 17, 18].”
Advances in Very Deep Convolutional Neural Networks for LVCSR (2016, methodology): “We performed stochastic sequence training as opposed to Hessian-Free sequence training [29] which was used in our baselines [6, 7].”
Joint training of convolutional and non-convolutional neural networks (2014, methodology): “The models were then retrained with Hessian-free sequence training as described in [16].”
Restructuring of deep neural network acoustic models with singular value decomposition (2013, methodology): “Neural networks used in acoustic models in ASR have usually been trained to perform frame classification with cross-entropy criterion or perform sequential training in recent studies [8][9].”
On rectified linear units for speech processing (2013, background): “This challenge continued to motivate researchers to design better unsupervised algorithms [3, 4] and better optimization methods for training deep neural nets [5, 6].”
Investigations on hessian-free optimization for cross-entropy training of deep neural networks (2013, methodology): “In Kingsbury’s paper [8], HF has only been applied to sequence training, which differs strongly from CE training, because it is initialized with a very good model that is obtained from CE training.”
A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion (2013, background): “The deep neural network (DNN) is an emerging technology that has recently demonstrated dramatic success in speech feature extraction and recognition, scaling very well from small [8][26][27][28] to medium [3][4][19][36] and to large [2][6][17] [21][34][32][37] tasks.”
Low-rank matrix factorization for Deep Neural Network training with high-dimensional output targets (2013, methodology): “After CE training, Hessian-free sequence-training [10] is performed to better adjust the weights for a sequence-level speech task.”
The IBM 2015 English conversational telephone speech recognition system (2015, methodology): “For both scenarios we generate numerator and denominator lattices with the initial joint model and optimize the lattice-based MBR loss using distributed hessian-free training [15].”