On the Optimization of a Synaptic Learning Rule

By Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, J. Gecsei. 2007

293 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “Past work has shown how to learn rules for learning (Schmidhuber, 1987; Bengio et al., 1992), train meta-agents that”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “The key idea underlying our method is to train the model’s initial parameters such that the model has maximal performance on a new task after the parameters have been updated through one or more gradient steps computed with a small amount of data from that new task.”
One-Shot Imitation Learning (2017, background): “Meta-learning has also been studied to discover neural network weight optimization algorithms [8, 9, 23, 50, 2, 31].”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “Few-shot learning [25, 35, 46, 53] has recently been primarily addressed in the meta-learning framework [57, 71, 80], where a model learns an update rule for the parameters of a base-learner model [6, 7, 72] through a sequence of training episodes [79, 88].”
Meta-learning with differentiable closed-form solvers (2018, background): “Another popular approach to meta-learning is to interpret the gradient update of SGD as a parametric and learnable function [16] rather than a fixed ad-hoc routine.”
Rewriting a Deep Generative Model (2020, background): “Examples include domain adaptation [66], zero-shot or few-shot learning [71,51], model pre-training and feature learning [17,82,78], and meta-learning [8,4,23].”
Recasting Gradient-Based Meta-Learning as Hierarchical Bayes (2018, methodology): “, 2017), or formulating the problem as one of learning to learn, or meta-learning (e.g., Schmidhuber, 1987; Bengio et al., 1991; Schmidhuber, 1992; Bengio et al., 1992).”
Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm (2017, methodology): “A common approach to meta-learning is to train a recurrent or memory-augmented model such as a recurrent neural network to take a training dataset as input and then output the parameters of a learner model (Schmidhuber, 1987; Bengio et al., 1992; Li & Malik, 2017a; Andrychowicz et al., 2016).”
Meta-learning autoencoders for few-shot prediction (2018, background): “This brings the key innovation of MeLA: for a class of tasks, MeLA does not attempt to learn a single good initialization for multiple tasks as in [7], or learn an update function [19, 2, 1], or learn an update function together with a single good initialization [13, 17].”
Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “Moreover, in the biological evolution of intelligent and complex forms of life, it is not clear yet which stepping stones were more challenging for natural evolution to overcome (Stanley and Lehman, 2015).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, methodology): “Recently popular approaches for ”meta-learning” (Schmidhuber, 1987; Bengio et al., 1992) have come to leverage protocols for what is called meta-training and meta-testing that also align with this assumption of a stationary task distribution p(z) (Wang et al.”
Incremental Meta-Learning via Indirect Discriminant Alignment (2020, methodology): “IML is not merely meta-training [31, 3, 35, 6, 25, 19] done by processing the training set in chunks: In IML, the meta-training set keeps changing, and we require both the performance of the model for the new tasks, as well as the base model, to improve.”
Learning to Learn for Global Optimization of Black Box Functions (2016, background): “For instance, one can learn to learn by gradient descent by gradient descent, or learn local Hebbian updates by gradient descent [Andrychowicz et al., 2016, Bengio et al., 1992].”
Watch, Try, Learn: Meta-Learning from Demonstrations and Reward (2019, background): “Learning to learn, or meta-learning, has a long-standing history in the machine learning literature [41, 33, 1, 15].”
Meta-Transfer Learning for Few-Shot Learning (2018, neutral): “In contrast to data-augmentation methods, meta-learning is a task-level learning method [2, 32, 51].”
Learned Optimizers that Scale and Generalize (2017, neutral): “A series of papers from Bengio et al. (1990; 1992; 1995) presents methods for learning parameterized local neural network update rules that avoid back-propagation.”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, background): “over structure that enables fast, few-shot learning in new tasks or environments. Our approach builds on techniques for meta-learning or learning-to-learn (Thrun & Pratt, 1998; Schmidhuber, 1987; Bengio et al., 1992; Naik & Mammone, 1992). However, most recent metalearning work considers a s”
Online Meta-Learning (2019, neutral): “Meta-learning: Prior meta-learning works have proposed learning an update rule or optimizer for fast adaptation (Hochreiter et al., 2001; Bengio et al., 1992; Andrychowicz et al., 2016; Li & Malik, 2017; Ravi & Larochelle, 2017) by having one model output the weights of another (Ha et al., 2017),…”