Deep Sequential Neural Networks

By Ludovic Denoyer. 2014

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Yoshua Bengio
2. Quoc V. Le
3. Marc'Aurelio Ranzato
4. Ruslan Salakhutdinov
5. Oriol Vinyals
6. Yann LeCun
7. Koray Kavukcuoglu
8. Geoffrey E. Hinton

Citations in the corpus

• Unified Scaling Laws for Routed Language Models (2022, methodology): “However this absence of bias comes with complications, especially the high variance of the gradient [Rosenbaum et al., 2019, Denoyer and Gallinari, 2014].”
• Gradient Episodic Memory for Continual Learning (2017, methodology): “This relates to methods that leverage a modular structure of the network with primitives that can be shared across tasks [Rusu et al., 2016, Fernando et al., 2017, Aljundi et al., 2016, Denoyer and Gallinari, 2015, Eigen et al., 2014].”
• Efficient Continual Learning with Modular Networks and Task-Driven Priors (2020, positive): “It is based on a modular neural network architecture (Eigen et al., 2014; Denoyer & Gallinari, 2015; Fernando et al., 2017; Li et al., 2019) with a novel task-driven prior (§4).”
• On Anytime Learning at Macroscale (2021, background): “The main issue is that MoE are notoriously harder to train (Eigen et al., 2014; Denoyer & Gallinari, 2015; Lepikhin et al., 2020).”