On rectified linear units for speech processing

By Matthew D. Zeiler, Marc'Aurelio Ranzato, Rajat Monga, Mark Z. Mao, Ke Yang, Quoc V. Le, Patrick Nguyen, Andrew W. Senior, Vincent Vanhoucke, Joseph Dean and 1 more. 2013

545 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification (2015, methodology): “In the speech recognition research of [34], the deep models degrade when using more than 8 hidden layers (all being fc).”
TensorFlow: A system for large-scale machine learning (2016, neutral): “In recent years, machine learning has driven advances in many different ﬁelds [3, 5, 23, 24, 30, 27, 40, 45, 48, 50, 55, 68, 69, 73, 76].”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “In contrast to the feed-forward AEs of [14, 25, 26], we incorporate temporal context by using a convolutional encoder and deconvolutional decoder in our AE.”
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain (2015, background): “Deep learning models are revolutionizing many fields such as computer vision [27, 23, 44], and speech and language processing [31, 47], and have achieved the status as the go-to state-of-theart techniques for many machine learning tasks.”
Expert Gate: Lifelong Learning with a Network of Experts (2016, methodology): “The encoding step consists of one fully connected layer followed by ReLU [37].”
A Simple Way to Initialize Recurrent Networks of Rectified Linear Units (2015, background): “…for the success of LSTMs. Recent research on deep feedforward networks has also produced some impressive results [19, 3] and there is now a consensus that for deep networks, rectiﬁed linear units (ReLUs) are easier to train than the logistic or tanh units that were used for many years [27, 40].”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, background): “There is also evidence that deep neural networks with rectiﬁed linear units can be trained successfully without pre-training (Zeiler et al., 2013).”
Fast, Compact, and High Quality LSTM-RNN Based Statistical Parametric Speech Synthesizers for Mobile Devices (2016, methodology): “The architecture of the acoustic LSTM-RNNs was 1 × 128-unit ReLU [39] layer followed by 3 × 128-cell LSTMP layers [40] with 64 recurrent projection units with a linear recurrent output layer [9].”
First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs (2014, background): “Recent work shows that with rectiﬁer nonlinearities DNNs can perform well in DNN-HMM systems without suffering from vanishing gradient problems during optimization (Dahl et al., 2013; Zeiler et al., 2013; Maas et al., 2013).”
A segmental framework for fully-unsupervised large-vocabulary speech recognition (2016, methodology): “The model signiﬁcantly outperformed the top-down GMM-UBM [32] and stacked AEs [28,29] in an intrinsic evaluation: isolated word discrimination.”
Norm-Based Capacity Control in Neural Networks (2015, methodology): “We will focus mostly on the hinge, or RELU (REctified Linear Unit) activation, which is currently in popular use (Nair and Hinton, 2010; Bordes and Bengio, 2011; Zeiler et al., 2013), σRELU(z) = [z]+ = max(z, 0).”
Analysis of function of rectified linear unit used in deep learning (2015, background): “To avoid this problem, Zieler et al. proposed the rectified linear unit (ReLU) [16], [7].”
Compressing deep neural networks using a rank-constrained topology (2015, methodology): “The baseline DNN consists of 3 fully-connected hidden layers with 128 nodes each, with a rectified linear unit (ReLU) activation function on each of the hidden layer nodes [19, 20].”
Unsupervised neural network based feature extraction using weak top-down constraints (2015, background): “While standard stacked AEs trained on speech (such as those of [9, 10]) use the same feature frame(s) as input and output, the correspondence AE uses weak top-down constraints in the form of (discovered) word pairs to have input and output frames from different instances of the same word.”
Automatic gain control and multi-style training for robust small-footprint keyword spotting with deep neural networks (2015, methodology): “We use rectified linear unit (ReLU) activation functions for the hidden layers [17].”
Deep neural network based instrument extraction from music (2015, methodology): “DNNs with ReLU activation functions have shown very good results, see for example [13] [14], and, as each layer has L hidden units, this activation function will allow us to use a layer-wise least squares initialization as will be shown in Sec.”