Randomized Block Coordinate Descent for Online and Stochastic Optimization

By Huahua Wang, Arindam Banerjee. 2014

36 citations recorded by the source corpus.

Two types of low cost-per-iteration gradient descent methods have been extensively studied in parallel. One is online or stochastic gradient descent (OGD/SGD), and the other is randomzied coordinate descent (RBCD). In this paper, we combine the two types of methods together and propose online randomized block coordinate descent (ORBCD). At each iteration, ORBCD only computes the partial gradient of one block coordinate of one mini-batch samples. ORBCD is well suited for the composite minimization problem where one function is the average of the losses of a large number of samples and the other is a simple regularizer defined on high dimensional variables. We show that the iteration complexity of ORBCD has the same order as OGD or SGD. For strongly convex functions, by reducing the variance of stochastic gradients, we show that ORBCD can converge at a geometric rate in expectation, matching the convergence rate of SGD with variance reduction and RBCD.

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

Federated Optimization: Distributed Machine Learning for On-Device Intelligence (2016, background): “There already exist attempts at combining SVRG type algorithms with randomized coordinate descent [46, 97].”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, background): “There already exist attempts at combining SVRG type algorithms with randomized coordinate descent [84, 180].”
Accelerated Mini-batch Randomized Block Coordinate Descent Method (2014, methodology): “While this paper was under review, we learnt that a similar method was independently proposed by [17].”
FedBCD: A Communication-Efficient Collaborative Learning Framework for Distributed Features (2022, unknown): “…that the loss function satisﬁes the following: Note that the second assumption of A3, which we refer to as the averaged gradient Lipschitz smooth condition, is stronger than directly assuming Lipschitz smoothness, but it is still a rather standard assumption in SGD analysis; see, e.g., [27], [28].”