Understanding Machine Learning - From Theory to Algorithms

By Shai Shalev-Shwartz, Shai Ben-David. 2014

3,984 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistical Learning Theory

Citations in the corpus

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models (2021, methodology): “While in most cases full fine-tuning reaches nearly 100% train accuracy, we find that the generalization gap (Shalev-Shwartz and Ben-David, 2014)—the difference between training error and test error—is substantially smaller for the BitFit models.”
Combined Scaling for Zero-shot Transfer Learning (2021, background): “…et al., 2012, Theorem 3.1) for the completeness (the proof utilizes the nonnegativity of h to have a slightly tighter bound than Theorem 26.5 of Shalev-Shwartz and Ben-David, 2014): Lemma 4 Let H be a set of maps z (cid:55)→ h ( z ) such that h ( z ) ∈ [0 , λ ] for all z in its domain. i Then,…”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, methodology): “Regarding generalization, namely how diﬀerent in-sample and out-of-sample errors are, our work relies on classic bounds as those that can be found, for example, in [26].”
Adaptive Federated Learning in Resource Constrained Edge Computing Systems (2018, methodology): “In such cases, stochastic gradient descent (SGD) is often used [6], [7], [37], which uses the gradient computed on the loss function defined on a randomly sampled subset (referred to as a minibatch) of data to approximate the real gradient.”
Adaptive Personalized Federated Learning (2020, methodology): “Hence, as a high-level idea, we propose to view this as the risk in the transformed label space L D (cid:48) i (ˆ h ∗ i ), and then employ known bounds in the agnostic PAC learning Shalev-Shwartz and Ben-David (2014).”
Robust Large Margin Deep Neural Networks (2016, methodology): “The GE bounds in [30], [32], [33] suggest that the GE of a DNN is bounded only if the number of training samples grows with the DNN depth or size.”
The Price of Fair PCA: One Extra Dimension (2018, background): “The following well-known fact characterizes the solutions to this classic problem [e.g., Shalev-Shwartz and Ben-David, 2014].”
A Meta-Transfer Objective for Learning to Disentangle Causal Mechanisms (2019, background): “…and Chervonenkis, 1971), and since VC-dimension grows approximately linearly in the number of parameters in linear models and neural networks Shalev-Shwartz and Ben-David (2014), the learning curve on the transfer distribution will tend to improve faster for the model with the correct…”
Machine Unlearning (2019, background): “Our efforts fall under the realm of supervised machine learning [22].”
Generative Continual Concept Learning (2019, methodology): “Conditioned on having large enough number of labeled data points n1, the empirical risk would be a suitable function to estimate the real risk function, e = E(x,y)∼p(t)(x,y)(Ld(fθ(t)(x),y)) (Shalev-Shwartz and Ben-David 2014) as the Bayes optimal objective.”
Regularisation of neural networks by enforcing Lipschitz continuity (2018, methodology): “This is accomplished using robustness theory, rather than the tools more typically used in learning theoretic bounds, such as Rademacher complexity and VC dimensions (Shalev-Shwartz and Ben-David, 2014).”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, neutral): “We consider the error between τ̂ and τ∗ using standard arguments (Shalev-Shwartz & Ben-David, 2014; Bach, 2014), i.e., d (τ̂ , τ∗) := J (τ̂)− J (τ∗) .”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “As the model is trained with supervised learning, sample error can be quantiﬁed by standard PAC generalization bounds, which bound the difference in expected loss and empirical loss by a constant with high probability (Shalev-Shwartz & Ben-David, 2014).”
Adversarially Robust Generalization Requires More Data (2018, neutral): “[44] Shai Shalev-Shwartz and Shai Ben-David.”
Exploring Generalization in Deep Learning (2017, background): “Even without making an assumption on the precision of parameters, the VC dimension of feedforward networks can be bounded in terms of the number of parameters dim(w)[1, 3, 6, 23].”
Network Topology and Communication-Computation Tradeoffs in Decentralized Optimization (2017, methodology): “Comparing Theorems 7 and 8, and ignoring the similar terms involving the initial conditions, we see that the convergence bound gets multiplied by 1 / (1 − λ ) .”
On the Computational Efficiency of Training Neural Networks (2014, background): “However, it is also possible to show that for this to happen, the size of the network must be exponential in d (e.g. [19, Chapter 20]).”