RNNDROP: A novel dropout for RNNS in ASR

By Taesup Moon, Heeyoul Choi, Hoshik Lee, Inchul Song. 2015

114 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Light Gated Recurrent Units for Speech Recognition (2018, methodology): “Since extending standard dropout to recurrent connections hinders learning long-term dependencies, we followed the approach introduced in [60], [61], that tackles this issue by sharing the same dropout mask across all the time steps.”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, methodology): “(1) with q ( ω ) , with (cid:98) ω k ∼ q ( ω ) , i.e. by performing dropout at test time and averaging results ( MC dropout ).”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “We conjecture that identity-masking is more appropriate for RNNs, since it makes it easier for the network to preserve information from previous time-steps going forwards, and it facilitates, rather than hinders, the flow of gradient information going backwards.”
Recurrent Dropout without Memory Loss (2016, methodology): “Moon et al. (2015) make the same observation and propose to use per-sequence mask sampling as well.”
Contextualized Word Representations for Reading Comprehension (2017, methodology): “For any single sequence, the same hidden state dropout mask was used across timesteps (Moon et al., 2015; Semeniuta et al., 2016; Gal and Ghahramani, 2016).”
Recent Advances in Recurrent Neural Networks (2017, methodology): “A dropout specifically tailored to RNNs is introduced in [82], called RNNDrop.”
Twin Regularization for online speech recognition (2018, methodology): “Recurrent dropout was used as regularization technique [43, 44].”
Twin Networks: Matching the Future for Sequence Generation (2017, methodology): “Dropout in RNNs has been proposed in (Moon et al., 2015), and was later extended in (Semeniuta et al.”
Deep learning in bioinformatics (2016, methodology): “To enhance the capabilities of dropout, a new activation function, maxout [53], and a variant of dropout for RNNs called rnnDrop [54], have been proposed.”
Single-Channel Multi-Speaker Separation Using Deep Clustering (2016, methodology): “Applying recurrent dropout to the LSTM memory cells recently yielded performance improvements on phoneme and speech recognition tasks with BLSTM acoustic models [23].”
Scalable Bayesian Learning of Recurrent Neural Networks for Language Modeling (2016, background): “Future works include improving the testing efficiency for the large-scale RNNs, via learning a single neural network that approximates the model averaging result (Korattikara et al., 2015).”
Improving Speech Recognition by Revising Gated Recurrent Units (2017, methodology): “Since extending standard dropout to recurrent connections hinders learning long-term dependencies, we followed the approach introduced in [38], which tackles this issue by sharing the same dropout mask across all the time steps.”