Gaussian Process Regression with Mismatched Models
By Peter Sollich. 2001
50 citations recorded by the source corpus.
Learning curves for Gaussian process regression are well understood when the 'student' model happens to match the 'teacher' (true data generation process). I derive approximations to the learning curves for the more generic case of mismatched models, and find very rich behaviour: For large input space dimensionality, where the results become exact, there are universal (student-independent) plateaux in the learning curve, with transitions in between that can exhibit arbitrarily many over-fitting maxima; over-fitting can occur even if the student estimates the teacher noise level correctly. In lower dimensions, plateaux also appear, and the learning curve remains dependent on the mismatch between student and teacher even in the asymptotic limit of a large number of training examples. Learning with excessively strong smoothness assumptions can be particularly dangerous: For example, a student with a standard radial basis function covariance function will learn a rougher teacher function only logarithmically slowly. All predictions are confirmed by simulations.
Subfields: Artificial Intelligence, Statistical and Nonlinear Physics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Frank Hutter
- Aaron Klein
- Stefan Falkner
- Matthieu Wyart
- Alessandro Favero
- Francesco Cagnetta
- Philipp Hennig
- Simon Bartels
In Statistical and Nonlinear Physics
- Abdulkadir Canatar
- Frank Hutter
- Aaron Klein
- Alessandro Favero
- Stefan Falkner
- Simon Bartels
- Francesco Cagnetta
- Matthieu Wyart
Citations in the corpus
- Fast Bayesian Optimization of Machine Learning Hyperparameters on Large Datasets (2016, background): “This might indicate a certain misfit between the true objective and the space of functions the GP can model [40].”
- Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks (2020, methodology): “The match to NTK training is excellent.”
- Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks (2020, methodology): “Here we derive the solution to the PDE in equation (17) of the main text by adapting the method used by (Sollich, 1999).”
- Locality defeats the curse of dimensionality in convolutional teacher–student scenarios (2021, background): “Note that in teacher-student settings for kernel ridgeless regression [31, 32, 4, 29] additional rigorous results are available: ◦ if teacher and student are isotropic kernels and data are sampled on a lattice Eq.”
- Fast Bayesian hyperparameter optimization on large datasets (2017, background): “This might indicate a certain misfit between the true objective and the space of functions the GP can model (Sollich, 2002).”