Language Modeling with Recurrent Highway Hypernetworks

By Joseph Suarez. 2017

38 citations recorded by the source corpus.

Subfields: Large Language Models, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. David R Ha
2. J. Urgen Schmidhuber
3. Justin Johnson
4. Li Fei-Fei
5. Maria R. Cervera
6. Johannes von Oswald
7. Alexander Meulemans
8. Benjamin Ehret

In Model Architectures

1. David R Ha
2. J. Urgen Schmidhuber
3. Li Fei-Fei
4. Razvan Pascanu
5. Justin Johnson
6. Raia Hadsell
7. Johannes von Oswald
8. Christian Henning

Citations in the corpus

• Recurrent World Models Facilitate Policy Evolution (2018, methodology): “Future work will explore replacing the VAE and MDN-RNN with higher capacity models [27, 89, 93, 97, 98], or incorporating an external memory module [19, 107], if we want our agent to learn to explore more complicated worlds.”
• World Models (2018, methodology): “Future work may explore replacing the small MDN-World RNN network with higher capacity models (Shazeer et al., 2017; Ha et al., 2016; Suarez, 2017; van den Oord et al., 2016; Vaswani et al., 2017), or incorporating an external memory module (Gemici et al., 2017), if we want our agent to learn to…”
• Meta-Learning with Warped Gradient Descent (2019, methodology): “For recurrent networks, we can use stacked LSTMs to interleave warped layers, as well as any type of HyperNetwork architecture (e.g. Ha et al., 2016; Suarez, 2017; Flennerhag et al., 2018) or partitioning of fast and slow weights (e.g. Mujika et al., 2017).”
• DDRprog: A CLEVR Differentiable Dynamic Reasoning Programmer (2018, methodology): “We use a recurrent highway network (RHN) (Zilly et al., 2016) as recommended by (Suarez, 2017) instead of an LSTM (Hochreiter & Schmidhuber, 1997) – both accept flat inputs.”
• Continual Learning in Recurrent Neural Networks with Hypernetworks (2020, background): “Another interesting direction is the use of a recurrent hypernetwork to generate timestep-specific weights in the main RNN [14, 51].”
• Continual learning in recurrent neural networks (2020, methodology): “An interesting direction is the use of a recurrent hypernetwork to generate timestep-specific weights in the main RNN (Ha et al., 2017; Suarez, 2017).”
• Principled Weight Initialization for Hypernetworks (2020, unknown): “…2017; Klocek et al., 2019; Serr ` a et al., 2019; Meyerson & Miikkulainen, 2019), continual learning (von Oswald et al., 2019), generative models (Suarez, 2017; Ratzlaff & Fuxin, 2019), ensemble learning (Kristiadi & Fischer, 2019), hyperpa-rameter optimization (Lorraine & Duvenaud, 2018), and…”