An Empirical Exploration of Recurrent Network Architectures

By Rafal Józefowicz, Wojciech Zaremba, Ilya Sutskever. 2015

1,805 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Exploring the Limits of Language Modeling (2016, neutral): “0 (Jozefowicz et al., 2015).”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, methodology): “To alleviate the difﬁculty of training RNNs due to vanishing and exploding gradients (Bengio et al., 1994), we use Gated Recurrent Units (GRUs) (Cho et al., 2014) which have been demonstrated to have good empirical performance (Chung et al., 2014; J´ozefowicz et al., 2015).”
Improved Variational Inference with Inverse Autoregressive Flow (2016, background): “ose to +1 or +2. This leads to an initial behaviour that updates z only slightly with each step of IAF. Such a parameterization is known as a 'forget gate bias' in LSTMs, as investigated by Jozefowicz et al. (2015). Perhaps the simplest special version of IAF is one with a simple step, and”
Deep Speech 2 : End-to-End Speech Recognition in English and Mandarin (2015, methodology): “ thousands of variations of LSTM and GRU architectures showed that a GRU is comparable to an LSTM with a properly initialized forget gate bias, and their best variants are competitive with each other [32]. We decided to examine GRUs because experiments on smaller data sets showed the GRU and LSTM re”
Light Gated Recurrent Units for Speech Recognition (2018, background): “On the other hand, computational efficiency is a crucial issue for RNNs and considerable research efforts have recently been devoted to the development of alternative architectures [7], [30], [46].”
Squeeze-and-Excitation Networks (2017, background): “While often computationally demanding, evolutionary search has had notable successes which include finding good memory cells for sequence models [27], [28] and learning sophisticated architectures for largescale image classification [29]–[31].”
Neural Architecture Search with Reinforcement Learning (2016, background): “On this task, LSTM architectures tend to excel (Zaremba et al., 2014; Gal, 2015), and improving them is difﬁcult (Jozefowicz et al., 2015).”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “roaches in designing architectures such as Neural Fabrics [48], DiffRNN [41], MetaQNN [3] and DeepArchitect [43]. A more ﬂexible class of methods for designing architecture is evolutionary algorithms [65, 16, 57, 30, 46, 42, 67], yet they have not had as much success at large scale. Xie and Yuille [”
Recurrent Highway Networks (2016, background): “It reduces model size for a fixed number of units and prevents an unbounded blow-up of state values leading to more stable training, but imposes a modeling bias which may be suboptimal for certain tasks (Greff et al., 2015; Jozefowicz et al., 2015).”
Visualizing and Understanding Recurrent Networks (2015, methodology): “e properties of its representations and predictions. Greff et al. (2015) recently conducted a comprehensive study of LSTM components. Chung et al. evaluated GRU compared to LSTMs Chung et al. (2014). Jozefowicz et al. (2015) conduct an automated architecture search of thousands of RNN architectures.”
LSTM: A Search Space Odyssey (2015, neutral): “[30].”
Large-Scale Evolution of Image Classifiers (2017, background): “t because they explore very different search spaces and have very different initial conditions (Table 2). Tangentially, there has also been neuro-evolution work on LSTM structure (Bayer et al., 2009; Zaremba, 2015), but this is beyond the scope of this paper. Also related to this work is that of Sax”
Learning to Communicate with Deep Multi-Agent Reinforcement Learning (2016, background): “5 and a learning rate of 5 10 4. The architecture makes use of rectiﬁed linear units (ReLU), and gated recurrent units (GRU) [19], which have similar performance to long short-term memory [20] (LSTM) [21, 22]. Unless stated otherwise we set ˙= 2, which was found to be essential for good performance.”
Weight Agnostic Neural Networks (2019, methodology): “In order to achieve state-of-the-art results, recent methods narrow the search space to architectures composed of basic building blocks with strong domain priors such as CNNs [64, 72, 89, 119], recurrent cells [48, 72, 119] and self-attention [100].”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, methodology): “NAS for RNN A few previous studies have attempted to optimize existing conventional recurrent cells and concluded that they are not necessarily optimal, moreover, the significance of their components is unclear [14, 22].”
AutoML-Zero: Evolving Machine Learning Algorithms From Scratch (2020, background): “Other AutoML studies similarly have found ways to constrain their search spaces to isolated algorithmic aspects, such as the learning rule used during backpropagation [2, 67], the gating structure of an LSTM [6, 34], or the data augmentation [17, 62]; in these works, all other algorithmic aspects re”
Bayesian Neural Architecture Search using A Training-Free Performance Metric (2020, background): “mizing the architecture of a deep neural network on a particular problem, including, for example, the type of activation functions, the number of hidden layers, and the number of units for each layer [4, 10, 22].”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, background): “While some works [22, 34, 91, 49] search RNN cells for language tasks, more works search good CNN architectures for image classification.”