Improving deep neural network acoustic models using generalized maxout networks

By Xiaohui Zhang, Jan "Yenda" Trmal, Daniel Povey, Sanjeev Khudanpur. 2014

315 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Don't Use Large Mini-Batches, Use Local SGD (2018, background): “…system resources (computation vs. communication), local SGD (a.k.a. local-update SGD, parallel SGD, or federated averaging) has recently attracted increased research interest (Mcdonald et al., 2009; Zinkevich et al., 2010; McDonald et al., 2010; Zhang et al., 2014; 2016; McMahan et al., 2017).”
Local SGD Converges Fast and Communicates Little (2018, background): “[22] for perceptron training (iterative parameter mixing), see also [7], [5, 46, 48] for the training of deep neural networks (model averaging) or in federated learning [23].”
Deep Learning: Methods and Applications (2014, background): “[441] generalize the above “maxout” units to two new types.”
Parallel training of Deep Neural Networks with Natural Gradient and Parameter Averaging (2014, methodology): “Backwards-in-time dependencies in our conventional neural net recipes, e.g. as reported in [15], include cepstral mean normalization (CMN), in which we subtract the mean of the input features; and fMLLR adaptation, also known as constrained MLLR adaptation [30], in which we use a baseline GMM…”
Letter-Based Speech Recognition with Gated ConvNets (2017, background): “HMM stands for Hidden Markov Model, CNN for ConvNet; when not speciﬁed, CNNs are 1D. pNorm is a particular non-linearity (Zhang et al., 2014).”
Deep Unfolding: Model-Based Inspiration of Novel Deep Architectures (2014, methodology): “…and (23), neglecting the schedule, and considering the reciprocal messages m k − 1 i j ( h j ) to be uniform, leads to the updates, This appears comparable to two commonly used forms: (29) is similar in spirit to softmaxout [25], and the max-product version (30) is similar in spirit to maxout [24].”
TTS Skins: Speaker Conversion via ASR (2019, methodology): “ree baselines. The ﬁrst is the WaveNet autoencoder voice conversion system of [19]. The second is a speaker conversion via PPG features marked as ’PPG’. PPG features were extracted using a p-norm DNN [37], as done by [16]. Finally, an additional PPG based speaker conversion system is trained by usin”
Convolutional deep maxout networks for phone recognition (2014, methodology): “found that p-norm pooling gives better results than those for max-pooling [13].”
The IBM 2015 English conversational telephone speech recognition system (2015, background): “Maxout networks for ASR have recently been investigated by several researchers, and found to produce significant gains when training data is limited [18], but negligible gains in our personal experience when the amount of training data exceeds approximately 100 hours.”
Ensemble-Compression: A New Method for Parallel Training of Deep Neural Networks (2016, methodology): “– MA-DNN refers the parallel DNN training framework with the aggregation by averaging model parameters [27, 20, 6, 26, 2, 3].”
Efficient Processing of Deep Neural Networks: A Tutorial and Survey (2017, background): “Finally, a non-linearity called maxout, which takes the max value of two intersecting linear functions, has shown to be effective in speech recognition tasks [44, 45].”
Acoustic Modelling from the Signal Domain Using CNNs (2016, methodology): “A normalization layer [21] is always added after the NIN nonlinearity to stabilize training.”
A comparison of neural network methods for unsupervised representation learning on the zero resource speech challenge (2015, unknown): “Our DNN architecture takes a 9-frame context window of MFCCs as input to 5 hidden layers of 5,000 units (2-norm maxout nonlinearity) followed by a linear bottleneck layer of 60 units (see [18] for details).”
Time delay deep neural network-based universal background models for speaker recognition (2015, background): “The hidden layers use the p-norm (where p = 2) activation function [19].”
Constructing long short-term memory based deep recurrent neural networks for large vocabulary speech recognition (2014, background): “Each layer in the “PNorm DNN” model had 800 pnorm units [31], where the hyperparameter p is set to 2, and the group size is set to 8.”
Improving speaker recognition performance in the domain adaptation challenge using deep neural networks (2014, methodology): “The training recipe is basically the same as described in [27], except that the fMLLR transform is removed both during training and testing (this was done to reduce computation).”
Scalable training of deep learning machines by incremental block training with intra-block parallel optimization and blockwise model-update filtering (2016, methodology): “Index Terms— Incremental block training, parallel optimization, blockwise model-update filtering, deep learning, LVCSR”