(Not) Bounding the True Error

By J. Langford, R. Caruana. 2001

149 citations recorded by the source corpus.

Subfields: Natural Language Processing, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Statistical Learning Theory

Citations in the corpus

Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, background): “See Langford & Caruana (2002) for the discussion around this technique.”
Exploring Generalization in Deep Learning (2017, background): “Others, including Langford and Caruana [13] and more recently Dziugaite and Roy [8], propose a PAC-Bayes analysis.”
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks (2017, methodology): “Langford and Caruana [6], and more recently Dziugaite and Roy [4] and Neyshabur et al. [12], used PAC-Bayes bounds to analyze generalization behavior in neural networks, evaluating the KL-divergence, “perturbation error” L [ f + ] − L [ f ] , or the entire bound numerically.”
Stronger generalization bounds for deep nets via a compression approach (2018, background): “A quantitative version of “ﬂatness” was suggested in [Langford and Caruana, 2001]: the net’s output is stable to noise added to the net’s trainable parameters.”
Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks (2018, background): “Generalization bounds for deep neural networks The phenomenon that deep neural networks generalize better than shallow neural networks have been observed in practice for a long time (Langford and Caruana, 2002).”
Bayesian Deep Learning and a Probabilistic Perspective of Generalization (2020, background): “In general, PAC-Bayes provides a compelling framework for deriving explicit non-asymptotic generalization bounds [31, 23, 7, 36, 37, 30, 17].”
User-friendly introduction to PAC-Bayes bounds (2021, methodology): “In order to optimize hyperparameters in PAC-Bayes bounds, Langford and Caruana [113] used a geometric grid Λ = {ek, k ∈ N} ∩ [1, n], the same choice was used later by Catoni [41, 43].”
Implicit Regularization in Deep Learning (2017, methodology): “By the definition of Q̃, we know that for any perturbation ũ: max x∈X |fw+ũ(x)− fw(x)|∞ < γ 4 (3.4.6) Therefore, the perturbation ũ can change the margin between two output units of fw by at most γ2 ; i.e. for any perturbation ũ drawn from Q̃: max i,j∈[k],x∈X |(|fw+ũ(x)[i]− fw+ũ(x)[j]|)− (|fw(”
Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach (2018, methodology): “This concept has also been widely studied (e.g., Langford, 2002; Langford & Caruana, 2002; Hinton & van Camp, 1993; Hochreiter & Schmidhuber, 1997; Baldassi et al., 2015; 2016; Chaudhari et al., 2017; Keskar et al., 2017; Dziugaite & Roy, 2017; Neyshabur et al., 2018).”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors (2017, background): “Their work builds on core insights by Langford & Caruana (2002), who computed nonvacuous bounds for neural networks ﬁve orders of magnitude smaller.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “Their work builds on the core insight demonstrated nearly 15 years ago by Langford and Caruana (2002), who computed nonvacuous bounds for neural networks ﬁve orders of magnitude smaller.”
SPECTRALLY-NORMALIZED MARGIN BOUNDS FOR NEURAL NETWORKS (2018, methodology): “Langford & Caruana (2001), and more recently Dziugaite & Roy (2017) and Neyshabur et al. (2017), used PAC-Bayes bounds to analyze generalization behavior in neural networks, evaluating the KL-divergence, “perturbation error” L[fw+u] − L[fw], or the entire bound numerically.”
Uniform convergence may be unable to explain generalization in deep learning (2019, background): “The ones that are small for realistic dataset sizes hold either only on a compressed network (Arora et al., 2018) or a stochastic network (Langford & Caruana, 2001) or a network that has been further modified via optimization or more than one of the above (Dziugaite & Roy, 2017; Zhou et al., 2019).”
The intriguing role of module criticality in the generalization of deep networks (2019, methodology): “We use a covering approach suggested by Langford & Caruana (2002).”
Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks (2019, background): “Existing generalization bounds for deep neural networks [29, 6, 27, 15, 13, 5, 24, 35, 28] based on uniform convergence usually cannot provide non-vacuous bounds [21, 13] in the over-parameterized regime.”
Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience (2019, background): “Langford & Caruana (2001); Dziugaite & Roy (2017) optimize the stochasticity and/or the weights of the network in order to numerically compute good (i.e., non-vacuous) generalization bounds on the stochastic network.”