Dropout Improves Recurrent Neural Networks for Handwriting Recognition

By Vu Tuan Dat Pham, Christopher Kermorvant, Jérôme Louradour. 2013

587 citations recorded by the source corpus.

Recurrent neural networks (RNNs) with Long Short-Term memory cells currently hold the best known results in unconstrained handwriting recognition. We show that their performance can be greatly improved using dropout - a recently proposed regularization method for deep architectures. While previous works showed that dropout gave superior performance in the context of convolutional networks, it had never been applied to RNNs. In our approach, dropout is carefully used in the network so that it does not affect the recurrent connections, hence the power of RNNs in modeling sequences is preserved. Extensive experiments on a broad range of handwritten databases confirm the effectiveness of dropout on deep architectures even when the network mainly consists of recurrent and shared connections.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Multi-task Sequence to Sequence Learning (2015, methodology): “(d) dropout is applied with probability of 0.2 over vertical connections (Pham et al., 2014), (e) we use SGD with a ﬁxed learning rate of 0.7, (f) input sequences are reversed, and lastly, (g) we use a simple ﬁnetuning schedule – after x epochs, we halve the learning rate every y epochs.”
Learning to Execute (2014, background): “Other interesting applications of recurrent neural networks includes speech recognition (Robinson et al., 1996; Graves et al., 2013), machine translation (Cho et al., 2014; Sutskever et al., 2014), handwriting recognition (Pham et al., 2013; Zaremba et al., 2014b), and many more. a r X i v : 1410 .”
Recurrent Neural Network Regularization (2014, methodology): “Independently of our work, (Pham et al., 2013) developed the very same RNN regularization method and applied it to handwriting recognition.”
Edinburgh Neural Machine Translation Systems for WMT 16 (2016, neutral): “Speciﬁcally, we sample a subset of the available target-side monolingual corpora, translate it automatically into the source side of the respective language pair, and then use this synthetic parallel data for training.”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, methodology): “In the case of POS tagging, the regularization is applied to θ e , and θ e is the embedding parameter after training the ﬁnal task in the top-most layer at the previous training epoch. δ is a hyperparameter.”
A Theoretically Grounded Application of Dropout in Recurrent Neural Networks (2015, neutral): “[9] assess dropout with handwriting recognition tasks.”
LSTM: A Search Space Odyssey (2015, methodology): “This includes handwriting recognition (Graves et al., 2009; Pham et al., 2013; Doetsch et al., 2014) and generation (Graves et al., 2013), language modeling (Zaremba et al., 2014) and translation (Luong et al., 2014), acoustic modeling of speech (Sak et al., 2014), speech synthesis (Fan et al.,…”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, methodology): “For the dual-layer LSTMs, we apply layer normalization [3] to all cells, with the dropout [27] with a rate of 0.”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “Zoneout is named after the popular dropout [Srivastava et al., 2014] regularization technique.”
Data Noising as Smoothing in Neural Network Language Models (2017, methodology): “For regularization, we apply feed-forward dropout (Pham et al., 2014) in combination with our noising schemes.”
Co-Occurrence Feature Learning for Skeleton Based Action Recognition Using Regularized Deep LSTM Networks (2016, background): “In order to preserve the ability of RNNs to model sequences, dropout applied only to the feed-forward (along layers) connections but not to the recurrent (along time) connections is proposed (Pham et al. 2014).”
Learning to Diagnose with LSTM Recurrent Neural Networks (2015, methodology): “Following the example of Zaremba et al. (2014) and Pham et al. (2014), we apply dropout to the non-recurrent connections only.”
Sequence-to-Sequence Learning as Beam-Search Optimization (2016, background): “Finally, it has been established that dropout (Sri-vastava et al., 2014) regularization improves the performance of LSTMs (Pham et al., 2014; Zaremba et al., 2014), and in our experiments we run beam search under dropout.”
Named Entity Recognition with Bidirectional LSTM-CNNs (2015, methodology): “We found applying dropout to the output nodes22 of each LSTM layer (Pham et al., 2014) was quite effective in reducing overfitting (Section 4.4).”
Recurrent Dropout without Memory Loss (2016, background): “Pham et al. (2013) and Zaremba et al. (2014) have shown that LSTMs can be effectively regularized when using dropout in forward connections.”
Key-Value Retrieval Networks for Task-Oriented Dialogue (2017, background): “We also added a bias of 1 to the LSTM cell forget gate in the style of (Pham et al., 2014).”