The Tradeoffs of Large Scale Learning

By L. Eon Bottou, Olivier Bousquet. 2007

1,747 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Scaling & Learning Theory

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “The dominant algorithms for training neural networks are based on mini-batch stochastic gradient descent (SGD, Robbins and Monro, 1951; Kiefer et al., 1952; Rumelhart et al., 1986; Bottou and Bousquet, 2008; LeCun et al., 2015).”
On Structured Prediction Theory with Calibrated Convex Surrogate Losses (2017, background): “data (x,y) ∼ D) to minimize the surrogate risk directly [6].”
Hogwild: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent (2011, background): “1 Introduction With its small memory footprint, robustness against noise, and rapid learning rates, Stochastic Gradient Descent (SGD) has proved to be well suited to data-intensive machine learning tasks [3,5,24].”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “We follow such an approach now, following the work in [20, 21].”
FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantization (2019, background): “It has been shown that for convex losses `, the population risk f is at most O(1/ √ nm) distant from the empirical risk L, uniformly and with high probability [Bottou and Bousquet, 2008].”
Robust and Communication-Efficient Collaborative Learning (2019, background): “In particular, it has been shown that when the loss function (cid:96) is convex, the difference between the population risk L and the empirical risk L N corresponding to N = mn samples with high probability is uniformly bounded by √ √ (Bottou and Bousquet, 2008).”
Better Mini-Batch Algorithms via Accelerated Gradient Methods (2011, methodology): “To understand the implication these bounds, we follow the approach described in [3, 12] to analyze large-scale learning algorithms.”
Adaptive Bound Optimization for Online Convex Optimization (2010, neutral): “ction 3. 1.2 The practical importance of adaptive regularization In the past few years, online algorithms have emerged as state-of-the-art techniques for solving largescale machine learning problems [Bottou and Bousquet, 2008, Zhang, 2004]. Two canonical examples of such large-scale learning problem”
Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, background): “An assessment of asymptotic performance of some optimization algorithms as learning algorithms in large-scale learning problems2 has been introduced in [13].”
Algorithmic recourse under imperfect causal knowledge: a probabilistic approach (2020, methodology): “We learn both the encoder and decoder parameters by maximising the evidence lower bound (ELBO) using stochastic gradient descend [9, 26, 27, 40].”
Counterfactual Prediction with Deep Instrumental Variables Networks (2016, background): “Our ﬁrst stage, ﬁtting F φ , is a standard density estimation problem and can be tuned to minimize the OOS deviance criterion min where f φ is either the probability mass function or density function associated with F φ , as appropriate.”
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning (2022, background): “The complexity of one training epoch for all classiﬁers in a CIL state is O (( ) 2 sd ) [3], with n - total number of classes in the dataset, d - dimensionality of features and s - samples per class.”
DeltaGrad: Rapid retraining of machine learning models (2020, background): “More recently a lot of work (see e.g., Bottou, 1998; 2003; Zhang, 2004; Bousquet & Bottou, 2008; Bottou, 2010; Bottou et al., 2018) focuses on empirical risk minimization.”
Batch Stationary Distribution Estimation (2020, background): “There is a complex trade-off between these factors [Bottou and Bousquet, 2008].”