Training and Analysing Deep Recurrent Neural Networks

By Michiel Hermans, Benjamin Schrauwen. 2013

491 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Deep Learning for IoT Big Data and Streaming Analytics: A Survey (2017, background): “However, considering the architecture and the functionality of RNNs, the hidden layers in RNNs are supposed to provide a memory instead of a hierarchical representation of features [40].”
An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling (2018, methodology): “Our work aims to compare generic convolutional and recurrent architectures on typical sequence modeling tasks that are commonly used to benchmark RNN variants themselves (Hermans & Schrauwen, 2013; Le et al., 2015; Joze-fowicz et al., 2015; Zhang et al., 2016).”
Deep Learning: Methods and Applications (2014, methodology): “A separate work on applying RNN to an LM with the unit of characters instead of words can be found in [153, 357].”
Architectural Complexity Measures of Recurrent Neural Networks (2016, background): “Some examples include Raiko et al. (2012); Graves (2013) who explored the use of skip connections; Hermans & Schrauwen (2013) who proposed the deep RNNs which are stacked RNNs with skip connections; Pascanu et al. (2013a) who pointed out the distinction of constructing a “deep” RNN from the view of…”
Deep Learning for Time Series Forecasting: The Electric Load Case (2019, background): “It has been empirically shown in several works that Deep RNNs are better to capture the temporal hierarchy exhibited by time-series then their shallow counterpart [54, 56, 57].”
Personal recommendation using deep recurrent neural networks in NetEase (2016, result): “Our DRNN is similar to the one studied in [8].”
Deep Recursive Neural Networks for Compositionality in Language (2014, background): “In order to combine these concepts, deep recurrent networks were proposed [10, 11, 12].”
Breaking Cryptographic Implementations Using Deep Learning Techniques (2016, background): “7, we explain how this time-dependency is used by the RNN during the profiling phase.”
Shortcut Sequence Tagging (2017, background): “Shortcut connections (shortcuts, or skip connections) enable unimpeded information ﬂow by adding direct connections across different layers (Raiko et al., 2012; Graves, 2013; Hermans and Schrauwen, 2013).”
On the Impact of Data Set Size in Transfer Learning Using Deep Neural Networks (2016, background): “RNNs have proven to be extremely powerful and can be used to do language translation [2], text prediction [27], image captioning [35], speech recognition [21], image generation [23] or even to evaluate short lines of programming code [75].”
Recurrent Memory Networks for Language Modeling (2016, methodology): “Previous work (Hermans and Schrauwen, 2013; Karpathy et al., 2015) studied this property of LSTMs by analyzing simple cases of closing brackets.”
Opinion Mining with Deep Recurrent Neural Networks (2014, methodology): “…RNN with multiple stacked hidden layers, are shown to naturally employ a temporal hierarchy with multiple layers operating at different time scales (Hermans and Schrauwen, 2013): lower levels capture short term interactions among words; higher layers reﬂect interpretations aggregated over longer…”
Recurrent Convolutional Network for Video-Based Person Re-identification (2016, methodology): “It is commonly accepted that the performance of deep networks is due to hierarchical feature extraction that takes place over many layers [11], therefore we use a CNN to pre-process each input image into a higher-level representation before the recurrent layer.”
From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning (2016, background): “Character-level language models are analyzed in Hermans and Schrauwen (2013) and Karpathy et al. (2015).”
Sequential Click Prediction for Sponsored Search with Recurrent Neural Networks (2014, background): “3) Recently, some research work (Hermans and Schrauwen 2013) has been done on Deep Recurrent Neural Networks (DRNN) and shows good results.”
Variational inference of latent state sequences using Recurrent Networks (2014, background): “Simultaneously, the theoretical efforts have lead to a better understanding of the learning dynamics [21] and the options of making RNNs more powerful have improved by means of architecture, learning and regularization [20, 13, 1, 3, 6].”