Learning curves for stochastic gradient descent in linear feedforward networks

By Zhengxue Li, Wei Wu, Guorui Feng, Huifang Lu

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “control parameter updates (Hochreiter et al., 2001; Andrychowicz et al., 2016), augment models with special memory mechanisms (Santoro et al.”
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, methodology): “This approach has been applied to learning to optimize neural networks (Hochreiter et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2017), as well as for learning dynamically changing recurrent neural networks (Ha et al.”
Learning to Optimize Neural Nets (2017, background): “Hochreiter et al. (2001) observed that a recurrent neural net with a particular setting of weights that takes a training example as input at each time step can be viewed as an online learning algorithm and learns an online learning algorithm by learning the weights of the recurrent neural net.”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “Another line of work (Hochreiter et al., 2001; Younger et al., 2001; Andrychowicz et al., 2016; Li & Malik, 2016) studies meta-learning over the optimization process.”
Neural Turing Machines (2014, background): “, 2014) and program search (Hochreiter et al., 2001b) (Das et al.”
What can the brain teach us about building artificial intelligence? (2016, background): “., 2016; Ravi and Larochelle, 2017; Vinyals et al., 2016; Wang et al., 2016;), another of Lake and colleagues’ key ingredients for humanlike intelligence. As recent work has shown (Duan et al., 2016; Hochreiter et al., 2001; Santoro et al., 2016; Wang et al., 2016;), this learning-to-learn mechanism”
One-Shot Imitation Learning (2017, background): “Meta-learning has also been studied to discover neural network weight optimization algorithms [8, 9, 23, 50, 2, 31].”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, background): “rate to a negative value for negative net inputs. Saturation means a small derivative of the activation function which decreases the variation and the information that is propagated to the next layer Hochreiter et al. (2001b); Hochreiter (1997; 1998). ELUs code the degree of presence of input concep”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “The meta-learner either learns to produce new parameters directly from the new data [9, 33, 56, 62, 64, 72, 73], or learns to produce an update rule to iteratively optimize the base learner to fit the new data [2, 6, 8, 38, 63, 97].”
Meta-learning with differentiable closed-form solvers (2018, background): “[25] observe that, because of the sequential nature of a learning algorithm, a recurrent neural network can be considered as a meta-learning system.”
Learning to Learn with Generative Models of Neural Network Checkpoints (2022, background): “They are typically trained with unrolled optimization (Hochreiter et al., 2001; Younger et al., 2001; Gregor & LeCun, 2010; Andrychowicz et al., 2016; Ravi & Larochelle, 2017; Wichrowska et al., 2017; Lv et al., 2017; Metz et al., 2019; 2022) or reinforcement learning (Li & Malik, 2016; 2017).”
Learning Transferable Architectures for Scalable Image Recognition (2017, background): “re notably below previous state-of-the-art (Table 2). The concept of having one neural network interact with a second neural network to aid the learning process, or learning to learn or meta-learning [23, 49] has attracted much attention in recent years [1, 62, 14, 19, 35, 45, 15]. Most of these app”
On First-Order Meta-Learning Algorithms (2018, methodology): “[8] who used LSTMs for next-step prediction and has been followed up by a burst of recent work, for example, Santoro et al.”
Meta-Learning for Semi-Supervised Few-Shot Classification (2018, background): “One approach to few-shot learning follows a form of meta-learning 1 (Thrun, 1998; Hochreiter et al., 2001), which performs transfer learning from a pool of various classiﬁcation problems generated from large quantities of available labeled data, to new classiﬁcation problems from classes unseen at…”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “The concept of using a Recurrent Neural Network for meta-learning has been attempted in the past, either via genetic programming or gradient descent (Schmidhuber, 1992; Hochreiter et al., 2001).”
Deep learning in neural networks: An overview (2014, methodology): “A successful metalearner (Hochreiter et al., 2001b) used an LSTM RNN to quickly learn a learning algorithm for quadratic functions (compare Sec.”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “lows for uniform credit assignment. The most commonly used LSTM memory cell architecture in the literature [39, 112] contains forget gates [31, 32] and peephole connections [30]. In our previous work [57, 53], we found that peephole connections are only useful for modeling time series, but not for l”
Meta-SGD: Learning to Learn Quickly for Few Shot Learning (2017, methodology): “Interestingly, [10] ﬁnd that LSTM performs best as meta-learner among various architectures of RNNs. [2] formulate LSTM as a generic, SGD-like optimizer which shows promising results compared to widely used hand-designed optimization algorithms.”