Learning internal representations

By Jonathan Baxter. 1995

Probably the most important problem in machine learning is the preliminary biasing of a learner's hypothesis space so that it is small enough to ensure good generalisation from reasonable training sets, yet large enough that it contains a good solution to the problem being learnt. In this paper a mechanism for {\em automatically} learning or biasing the learner's hypothesis space is introduced. It works by first learning an appropriate {\em internal representation} for a learning environment and then using that representation to bias the learner's hypothesis space for the learning of future tasks drawn from the same environment. An internal representation must be learnt by sampling from {\em many similar tasks}, not just a single task as occurs in ordinary machine learning. It is proved that the number of examples $m$ {\em per task} required to ensure good generalisation from a representation learner obeys $m = O(a+b/n)$ where $n$ is the number of tasks being learnt and $a$ and $b$ are constants. If the tasks are learnt independently ({\em i.e.} without a common representation) then $m=O(a+b)$. It is argued that for learning environments such as speech and character recognition $b\gg a$ and hence representation learning in these environments can potentially yield a drastic reduction in the number of examples required per task. It is also proved that if $n = O(b)$ (with $m=O(a+b/n)$) then the representation learnt will be good for learning novel tasks from the same environment, and that the number of examples required to generalise well on a novel task will be reduced to $O(a)$ (as opposed to $O(a+b)$ if no representation is used). It is shown that gradient descent can be used to train neural network representations and experiment results are reported providing strong qualitative support for the theoretical results.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Survey on Multi-Task Learning (2017, background): “, [49], [249], [250], [253], [254], [257], [258]) analyze different MTL models based on various analysis tools and the best convergence rate is Oð 1 ffiffiffiffiffiffiffi mn0 p Þ.”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, neutral): “Then, the model can be fine-tuned [8, 6, 22].”
A Model of Inductive Bias Learning (2000, methodology): “For more details see Baxter (1995b) and Baxter (1995a, chapter 4 ), where empirical results supporting the theoretical results presented here are also given.”
On the momentum term in gradient descent learning algorithms (1999, background): “The momentum term helps average out the oscillation along the short axis while at the same time adds up contributions along the long axis (Rumelhart et al., 1986).”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “Reuse was also exploited by sharing the lower layers of a network (and the representation of the input that they capture) across multiple tasks associated with diﬀerent outputs of the network (Caruana, 1993; Baxter, 1995, 1997).”
Meta-Learning: A Survey (2018, neutral): “odels can be used as a good initialization for the target model, yielding a pre-trained model which can then be further ﬁne-tuned using the available training data on t new (Thrun and Mitchell, 1995; Baxter, 1996; Bengio, 2012; Caruana, 1995). In some cases, the source network may need to be modiﬁed”
A Baseline for Few-Shot Image Classification (2019, background): “The meta-training loss is designed to make few-shot training efficient [15, 16, 17, 18].”
Deep Rewiring: Training very sparse deep networks (2017, methodology): “The error gradient k E X , Y ∗ ( θ ) can be computed using standard Error Backpropagation Neal (1992); Rumelhart et al. (1985).”
Bilevel Programming for Hyperparameter Optimization and Meta-Learning (2018, methodology): “4, by taking inspiration on early work on representation learning in the context of multi-task and meta-learning (Baxter, 1995; Caruana, 1998), we instantiate the framework for ML in a simple way treating the weights of the last layer of a neural network as the inner variables and the remaining…”
Multitask Sparsity via Maximum Entropy Discrimination (2011, background): “Multitask learning has many names and incarnations including learning-to-learn, meta-learning, lifelong learning, and inductive transfer (Baxter, 1995; Thrun and Pratt, 1997; Caruana, 1997; Thrun, 1995).”
Multi-task feature and kernel selection for SVMs (2004, methodology): “Multi-task learning or meta-learning leverages these many datasets synergistically, aggregating them and augmenting the effective size of the total training data (Baxter, 1995; Thrun & Pratt, 1997; Caruana , 1997).”
Can deep learning beat numerical weather prediction? (2021, background): “The invention of backpropagation in 1970 [5,6] led to a second wave of ML applications as it became possible to build more extensive NNs and train them to recognize nonlinear relationships in data (e.g. [ 7]).”
Optimal Decision Trees (1996, methodology): “Our global approach is analogous to the widely used back propagation algorithm for constructing neural networks [31].”
Planning simple trajectories using neural subgoal generators (1993, methodology): “With architecture 1, this is essentially done by back-propagating error signals (e.g. (Werbos,1974), (Parker, 1985), (LeCun, 1985), (Rumelhart et al., 1986)) through copies of the evaluatormodules down into the subgoal generator.”