Convex Optimization for Big Data: Scalable, randomized, and parallel algorithms for big data analytics

By Volkan Cevher, Stephen Becker, Mark W. Schmidt. 2014

308 citations recorded by the source corpus.

This article reviews recent advances in convex optimization algorithms for big data, which aim to reduce the computational, storage, and communications bottlenecks. We provide an overview of this emerging field, describe contemporary approximation techniques such as first-order methods and randomization for scalability, and survey the important role of parallel and distributed computation. The new big data algorithms are based on surprisingly simple principles and attain staggering accelerations even on classical problems.

Subfields: Artificial Intelligence, Computational Mechanics, Computational Theory and Mathematics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computational Theory and Mathematics

Citations in the corpus

Achieving Geometric Convergence for Distributed Optimization Over Time-Varying Graphs (2016, background): “The distributed algorithms designed over such special structure are usually fast in practice and mostly used in machine learning to handle big-data in a cluster of computers [6,8].”
DSA: Decentralized Double Stochastic Averaging Gradient Algorithm (2015, background): “Although we make no formal assumption, in cases of practical importance the total number of training samples =1 q n is very large, but the numbers of elements q n available at a speciﬁc node are moderate.”
Multi-Agent Distributed Lifelong Learning for Collective Knowledge Acquisition (2017, background): “Distributed Machine Learning: There has been a growing interest in developing scalable learning algorithms using distributed optimization [40], motivated by the emergence of big data [6], security and privacy constraints [37], and the notion of cooperative and collaborative learning agents [8].”
Bayesian computation: a summary of the current state, and samples backwards and forwards (2015, background): “…Bauschke and Combettes (2011) for the mathematics underpinning proximal optimisation algorithms, and to the recent tutorial papers by Combettes and Pesquet (2011), Cevher et al. (2014) and Parikh and Boyd (2014) for an overview of the ﬁeld and applications to signal processing and machine learning.”
NESTT: A Nonconvex Primal-Dual Splitting Method for Distributed and Stochastic Optimization (2016, background): “It arises frequently in applications such as machine learning and signal processing; see a recent survey [7].”
DEXTRA: A Fast Algorithm for Optimization Over Directed Graphs (2017, background): “D ISTRIBUTED computation and optimization have gained great interests due to their widespread applications in, e.g., large-scale machine learning [1], [2], model predictive con-trol [3], cognitive networks [4], [5], source localization [6], [7], resource scheduling [8], and message routing [9].”
DQM: Decentralized Quadratically Approximated Alternating Direction Method of Multipliers (2015, background): “The nodes’ goal is to ﬁnd the optimal argument of the global cost function Problems of this form arise in, e.g., decentralized control [2]– [4], wireless communication [5], [6], sensor networks [7]–[9], and large scale machine learning [10]–[12].”
Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 1 Low-Rank Tensor Decompositions (2016, unknown): “e random coordinate descent (RCD) scheme [195, 228], subgradient methods [196], alternating direction method of multipliers (ADMM) methods [31], and proximal gradient descent methods [213], (see also [40,124] and references therein). This monograph systematically introduces TN models and the associa”
Bayesian computation: a perspective on the current state, and sampling backwards and forwards (2015, background): “…and Com-bettes (2011) for the mathematics underpinning proximal optimisation algorithms, and to the recent tutorial papers by Combettes and Pesquet (2011), Cevher et al. (2014) and Parikh and Boyd (2014) for an overview of the ﬁeld and applications to signal processing and machine learning.”
2016 Ieee International Conference on Big Data (big Data) Consensus Optimization with Delayed and Stochastic Gradients on Decentralized Networks (unknown): “Distributed computing on networks is an emerging technology with extensive applications in big data analysis [1] and modern machine learning [6].”
A Framework for Parallel and Distributed Training of Neural Networks (2016, methodology): “Also in this case, even if problem (30) cannot be solved in closed form, it is possible to implement very fast and efficient algorithms for its solution, see, e.g., (Schmidt, 2010; Cevher et al., 2014).”
D-FW: Communication efficient distributed algorithms for high-dimensional sparse optimization (2016, background): “, big-data optimization [1] and large-scale sparse learning [2, 3].”
Proximal Splitting Algorithms: A Tour of Recent Advances, with New Twists. (2020, background): “Moreover, there are many avenues for extensions, including the use of variable stepsizes or metrics [22,52], the study of convergence rates and development of accelerated variants [104–108], or the design of randomized versions [109–113].”
Machine Learning and Integrative Analysis of Biomedical Big Data (2019, background): “Specifically, convex optimization, a key competent in training many ML and statistical models, is being reinvented for scalability and parallelism in the wake of big data [193].”
A Universal Primal-Dual Convex Optimization Framework (2015, background): “Processing the inclusion Ax´ b P K in (1) requires a significant computational effort in the largescale setting [4].”