Foundations of Machine Learning

By N. Nathani, Abhishek Singh. 2021

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

High-dimensional dynamics of generalization error in neural networks (2017, background): “…training a large non-linear neural network with a random first layer, the empirical Rademacher complexity of a classification problem (see e.g. Mohri, Rostamizadeh, and Talwalkar, 2012) is R(H) = 〈 sup h∈H 1 P P∑ µ=1 σµh(xµ) 〉 σ,x , (33) where H is the class of functions the network or…”
The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks (2020, background): “01 ‖NTK‖ ) so that we can better observe early time of training (similar to Nakkiran et al. [2019]).”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “Lemma 4 is a direct application of previous results (Bartlett and Mendelson, 2002; Mohri et al., 2012; Shalev-Shwartz and Ben-David, 2014) to our problem.”
Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “To ease the reading, we suggest non-expert readers to also consult elementary machine learning textbooks such as Bishop (2006), Schölkopf and Smola (2002), Mohri et al. (2012), and Murphy (2012).”
A Theory of Multiple-Source Adaptation with Limited Target Labeled Data (2020, background): “By standard learning-theoretic tools (Mohri et al., 2018), the generalization bound for this model can be stated as follows: for simplicity let the loss be the zeroone loss.”
Three Approaches for Personalization with Applications to Federated Learning (2020, background): “By standard learning-theoretic tools (Mohri et al., 2018a), the performance of this model can be bounded as follows: with probability at least 1 − δ , the minimizer of empirical risk L ̂ D 1 h satisﬁes ⎝ ⎠ where d is the VC-dimension of the hypothesis class H .”
Agnostic Federated Learning (2019, background): “Proof The proof is an extension of the standard proofs for Rademacher complexity generalization bounds (Koltchinskii and Panchenko, 2002; Mohri et al., 2018).”
Adaptive Personalized Federated Learning (2020, methodology): “…classiﬁcation problem: where the second term can be bounded by standard generalization bounds from agnostic PAC learning (e.g., please see Shalev-Shwartz and Ben-David (2014); Mohri et al. (2018)): Since we assume loss function is convex so we apply the Jensen’s inequality on the ﬁrst term in (29).”
AdaNet: Adaptive Structural Learning of Artificial Neural Networks (2016, background): “Because of this, while large models are more powerful, they also become increasingly more prone to overﬁtting.”
The Ideal Continual Learner: An Agent That Never Forgets (2023, background): “, Shalev-Shwartz & Ben-David (2014); Mohri et al. (2018) for some basics of statistical learning.”
Batch Stationary Distribution Estimation (2020, background): “It has been shown that the RHS of (22) is contractive (Sutton and Barto, 1998; Mohri et al., 2012), therefore, the ﬁx-point iteration, converges to the true ⌧ as t !”
Minimax Weight and Q-Function Learning for Off-Policy Evaluation (2019, neutral): “This is proved by Mercer’s theorem (Mohri et al., 2012).”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “Depending on our policy representation, V may be the VC dimension [85] or distribution-dependent measures such as (empirical) Rademacher complexity [4, 41].”
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks (2019, neutral): “(Mohri et al., 2012)): Theorem B.”
A Theoretical Analysis of Contrastive Unsupervised Representation Learning (2019, methodology): “First, we use the classical bound for the generalization error in terms of the Rademacher complexity of the function class (see (Mohri et al., 2018) Theorem 3.1).”
Spectrally-normalized margin bounds for neural networks (2017, background): “Since (cid:96) γ has range [0 , 1] , it follows by standard properties of Rademacher complexity (Mohri et al., 2012, Theorem 3.1) that with probability at least 1 − δ , every f ∈ F satisﬁes The bound now follows by applying Lemma A.4 to the left hand side.”
Interpolation Consistency Training for Semi-Supervised Learning (2019, methodology): “Interpolation-based regularizers (Tokozume et al., 2018; Verma t al., 2018; Zhang et al., 2018) have been recently proposed or supervised learning, achieving state-of-the-art performances cross a variety of tasks and network architectures. While okozume et al. (2018) and Zhang et al. (2018) proposed”
Generalization in Deep Learning (2017, methodology): “Some classical theory work attributes generalization ability to the use of a low-capacity class of hypotheses (Vapnik, 1998; Mohri et al., 2012).”
Fantastic Generalization Measures and Where to Find Them (2019, methodology): “Using these bounds on d (cid:48) , a i and q i the equivalent network, we can bound the VC dimension as follows: For binary classiﬁers, generalization error can be in terms of Rademacher complexity (Mohri et al., 2012) which in turn can be bounded by 72 VC /m (Kontorovich, 2016).”
SOLA: Continual Learning with Second-Order Loss Approximation (2020, background): “ rank approximation. The reason for this simpliﬁcation is that our focus is the optimization aspect of the problem, while the generalization aspect can be tackled by tools such as uniform convergence [30]. As discussed, during the training of the last task, we have access to the approximate loss fun”