Optimal Distributed Online Prediction

By Ofer Dekel, Ran Gilad-Bachrach, Ohad Shamir, Lin Xiao. 2011

69 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Better Mini-Batch Algorithms via Accelerated Gradient Methods (2011, methodology): “In this more general case we let domain W be some closed convex set of a Banach space equipped with norm (cid:107)·(cid:107) .”
Optimal Distributed Online Prediction Using Mini-Batches (2010, background): “This direction has been further explored in Dekel et al. (2011).”
Communication-Efficient Distributed Dual Coordinate Ascent (2014, background): “In the machine learning setting, [26] and the later work of [27] have provided additional insights into these types of methods.”
Fundamental Limits of Online and Distributed Algorithms for Statistical Learning and Estimation (2013, background): “Some examples include: • Communication constraints in distributed learning: There has been much work in recent years on learning when the training data is distributed among several machines (with [14, 2, 28, 47, 25, 31, 9, 17, 38] being just a few examples).”
Feature space maximum a posteriori linear regression for adaptation of deep neural networks (2014, methodology): “The compromise between the two, the mini-batch SGD [34], is more widely used, as the reasonable size of mini-batches makes all the matrices ﬁt into the GPU memory, leading to a computationally efﬁcient learning process.”
Stochastic Mirror Descent: Convergence Analysis and Adaptive Variants via the Mirror Stochastic Polyak Stepsize (2021, methodology): “Under smoothness, similar improvements can be made (Bubeck, 2014; Dekel et al., 2012).”
Solving Large Scale Linear SVM with Distributed Block Minimization (2011, background): “[5, 11, 13, 21] perform parallelization over examples in the primal space, while DBM does this in the dual one.”
Accelerated Mirror Descent in Continuous and Discrete Time (2015, methodology): “The mirror descent method also provides an important generalization of the gradient descent method to non-Euclidean geometries, as discussed in [19, 3], and has many applications in convex optimization [6, 5, 12, 15], as well as online learning [9, 11].”
Rapid adaptation for deep neural networks through multi-task learning (2015, methodology): “Mini-batchstochastic gradient descent (SGD) [29], with a reasonable size of mini-batches to make all matrices ﬁt into the GPU memory, was used to update all neural network parameters during training.”
Adaptation, Learning, and Optimization over Networks (2014, background): “This scenario is also common in machine learning problems [4, 37, 85, 193, 234, 240] when data samples at the various agents are generated by a common distribution parameterized by some vector, wo.”
A Distributed Frank-Wolfe Algorithm for Communication-Efficient Sparse Learning (2014, methodology): “From a more practical view, a lot of research has gone into developing scalable algorithms with small communication and synchronization overhead (see for instance [8, 9, 10, 11, 12, 13, 14] and references therein).”
Crowd-ML: A Privacy-Preserving Learning Framework for a Crowd of Smart Devices (2015, methodology): “Crowd-ML also builds on recent advances in incremental distributed learning [20], [21], which show that a near-optimal convergence rate is achievable despite communication delays.”
Maximum a posteriori adaptation of network parameters in deep models (2015, methodology): “Mini-batch stochastic gradient descent (SGD) [32], with a reasonable size of mini-batches to make all matrices fit into the GPU memory, was used to update all neural parameters during training.”
Towards a unified architecture for in-RDBMS analytics (2012, background): “There has been a good deal of work in the machine learning community to create several clever parallelization schemes for IGD [12,18,20,29,53].”
Excess-Risk of Distributed Stochastic Learners (2013, background): “ect is related to the fact that the lack of strong-convexity causes a non-cooperative solution to at best have O(1= p i) performance for iiterations. It is unclear if the algorithms proposed in [10], [11] can converge at the rate O(1=Ni) if the risk function is strongly-convex. Furthermore, while di”
Diffusion Adaptation Strategies for Distributed Optimization and Learning Over Networks (2011, background): “Assume the individual cost Jl(w) at node l can be expressed as the expected value of a certain loss function Ql(·, ·), i.e., Jl(w) = E{Ql(w,xl,i)}, where the expectation is with respect to the randomness in the data samples {xl,i} that are collected at node l at time i.”
Adaptive Networks (2014, background): “This scenario is also common in machine learning problems [22], [23], [62]–[64] when data samples at the various agents are generated by a common distribution parameterized by some vector w o .”