Learning from hints in neural networks

By Yaser S. Abu-Mostafa. 1990

Abstract Learning from examples is the process of taking input-output examples of an unknown function ƒ and infering an implementation of ƒ. Learning from hints allows for general information about ƒ to be used instead of just input-output examples. We introduce a method for incorporating any invariance hint about ƒ in any descent method for learning from examples. We also show that learning in a neural network remains NP-complete with a certain, biologically plausible, hint about the network. We discuss the information value and the complexity value of hibts.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “The easiest way to do this is through hints [Abu-Mostafa, 1990], i.e. directly training the model to predict the most important features.”
Dropout Training as Adaptive Regularization (2013, background): “Dropout training falls into the broader category of learning methods that artiﬁcially corrupt training data to stabilize predictions [2, 4, 5, 6, 7].”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “Early works sometimes referred to this as including “hints” (Suddarth and Kergosien, 1990; Abu-Mostafa, 1990) to improve learning.”
Occam's Two Razors: The Sharp and the Blunt (1998, background): “38 Domingos Similar remarks apply to the more recent work of MacKay (1992). The "Occam factors" that appear his evidence framework penalize model structures with many parameters, as opposed to models, and can also lead to suboptimal choices.”
A Practical Bayesian Framework for Backprop Networks (1991, background): “Some of the `hints' in (Abu{Mostafa, 1990b) also fall into the category of additive weight{dependent energies.”
Bayesian Methods for Adaptive Models (2011, methodology): “Some of the ‘hints’ in [2] also fall into the category of additive weight–dependent energies.”
Adaptive stochastic resonance (1998, background): “These rules o er a simple way to endow a fuzzy approximator with prior knowledge or \hints" [1, 2] that can improve how well a fuzzy system approximates a function or how well it generalizes from training samples [197].”
Combining Symbolic and Neural Learning (1994, background): “One can think of this preexisting information as prior knowledge about the task at hand, and the question is: how can neural networks effectively use these "hints" (Abu-Mostafa, 1990)? One answer, the KBANN approach (Towell, Shavlik, & Noordewier, 1990; Towell, 1992), creates knowledge-based artific”
A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling (1997, background): “Several authors have made empirical studies of the idea that learning multiple related tasks should improve performance, see e.g. (Caruana, 1993, Abu-Mostafa, 1989, Mitchell & Thrun, 1994).”
Sequence Learning (2001, background): “One possibility is to give some hints to the network on specific properties of the desired function (see for example [1] [6] [58] [3] [2] [5] [4]) or to insert prior knowledge in form of rules into the neural network and then to train it using a standard learning algorithm (for the case of dynamical”
A Practical Bayesian Framework for Backpropagation Networks (1992, background): “Some of the "hints" in Abu-Mostafa (1990b) also fall into the category of additive weight-dependent energies.”
Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference (2001, methodology): “This can be seen as similar to the use of hints (Abu-Mostafa, 1990) and should help to make the problem less ill-posed.”
Data Augmentation via Levy Processes (2016, unknown): “One popular strategy for encoding such a belief is data augmentation: generating additional pseudo-examples or “hints” by applying label-invariant transformations to training examples’ features (Abu-Mostafa, 1990; Schölkopf et al., 1997; Simard et al., 1998).”