A Model of Inductive Bias Learning

By Jonathan Baxter. 2000

1,319 citations recorded by the source corpus.

A major problem in machine learning is that of inductive bias: how to choose a learner's hypothesis space so that it is large enough to contain a solution to the problem being learnt, yet small enough to ensure reliable generalization from reasonably-sized training sets. Typically such bias is supplied by hand through the skill and insights of experts. In this paper a model for automatically learning bias is investigated. The central assumption of the model is that the learner is embedded within an environment of related learning tasks. Within such an environment the learner can sample from multiple tasks, and hence it can search for a hypothesis space that contains good solutions to many of the problems in the environment. Under certain restrictions on the set of all hypothesis spaces available to the learner, we show that a hypothesis space that performs well on a sufficiently large number of training tasks will also perform well when learning novel tasks in the same environment. Explicit bounds are also derived demonstrating that learning multiple tasks within an environment of related tasks can potentially give much better generalization than learning a single task.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “Formally, language modeling induces a hypothesis space H that should be useful for many other NLP tasks (Vapnik and Kotz, 1982; Baxter, 2000).”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “Our motivation for multi-task training stems from theoretical insights presented in Baxter et al. (2000). We refer readers to that work for a detailed discussion of results, but the conclusions most relevant to this discussion are (i) that learning multiple related tasks jointly results in good gene”
An Overview of Multi-Task Learning in Deep Neural Networks (2017, background): “[Baxter, 2000] argues only theoretically that related tasks share a common optimal hypothesis class, i.e. have the same inductive bias.”
An Overview of Multi-task Learning (2018, neutral): “ThefirstgeneralizationboundforMTLisderived in[133]forageneralMTLmodel.”
Fine-tuned Language Models for Text Classification (2018, background): “Formally, the pretraining task should allow us to induce a hypothesis space H that will be useful for many other NLP tasks (Vapnik and Kotz, 1982; Baxter, 2000).”
What can the brain teach us about building artificial intelligence? (2016, background): “ems, learning-to-learn can occur through the sharing of features between the models learned for old objects (or old tasks) and the models learned for new objects (or new tasks) (Anselmi et al., 2016; Baxter, 2000; Bottou, 2014; Lopez-Paz, Bottou, Scholkopf, & Vapnik, 2016; Salakhutdinov, Torralb”
A theory of learning from different domains (2010, background): “Although these bounds have a more limited scope than ours, they can sometimes yield useful results even when the optimal predictors for each task (or domain in the case of Daumé 2007) are quite different (Baxter 2000; Ando and Zhang 2005).”
Advances and Open Problems in Federated Learning (2019, background): “In the standard learning-to-learn (LTL) setup [52], one has a meta-distribution over tasks, samples from which are used to learn a learning algorithm, for example by finding a good restriction of the hypothesis space.”
Time-series forecasting with deep learning: a survey (2020, background): “By incorporating bespoke architectural assumptions—or inductive biases [14]—that reflect the nuances of underlying datasets, deep neural networks are able to learn complex data representations [15], which alleviates the need for manual feature engineering and model design.”
Differentially Private Meta-Learning (2019, methodology): “Our learning result follows the setup of Baxter (2000), who formalized the LTL problem as using task-distribution samples P1, . . . ,PT ∼ Q from some meta-distribution Q and samples indexed by i = 1, . . . ,m from those tasks to improve performance when a new task P is sampled fromQ and we draw m…”
Adaptive Gradient-Based Meta-Learning Methods (2019, methodology): “In contrast, the ARUBA framework can handle general classes of meta-algorithms, which leads not only to new and improved results in static, dynamic, and statistical settings but also to signiﬁcantly more practical LTL methods.”
Semi-Cyclic Stochastic Gradient Descent (2019, background): “In order to do so, it is necessary to target a particular way in which tasks are related (Baxter, 2000; Ben-David & Schuller, 2003).”
Gradient Episodic Memory for Continual Learning (2017, background): “This learning setting led to implementations [Carlson et al., 2010, Ruvolo and Eaton, 2013, Ring, 1997], and theoretical investigations [Baxter, 2000, Balcan et al., 2015, Pentina and Urner, 2016], although the latter ones have been restricted to linear models.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “One early assumption on the task relationship is given by Baxter (2000) in the context of learning to learn, who assumed a statistical data and task generation model such that, a hypothesis space that contains good solutions for a new task can be learned from sufficiently many samples of old tasks. ”
Model Zoo: A Growing Brain That Learns Continually (2021, background): “If it seeks a small average population risk it may do so by minimizing the average empirical risk As Baxter (2000) shows, under very general conditions, if then we have e ¯ P (¯ h ) ≤ e ¯ S (¯ h ) + (cid:15) for any ¯ h ∈ H n .”