Stochastic Gradient Descent on Riemannian Manifolds

By Silvere Bonnabel. 2011

716 citations recorded by the source corpus.

Stochastic gradient descent is a simple approach to find the local minima of a cost function whose evaluations are corrupted by noise. In this paper, we develop a procedure extending stochastic gradient descent algorithms to the case where the function is defined on a Riemannian manifold. We prove that, as in the Euclidian case, the gradient descent algorithm converges to a critical point of the cost function. The algorithm has numerous potential applications, and is illustrated here by four examples. In particular a novel gossip algorithm on the set of covariance matrices is derived and tested numerically.

Subfields: Artificial Intelligence, Numerical Analysis, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Numerical Analysis

In Statistics and Probability

Citations in the corpus

Continual Learning in Low-rank Orthogonal Subspaces (2020, positive): “The optimization of a differentiable cost function over a Stiefel manifold has been extensively studied in literature [Absil et al., 2009, Bonnabel, 2013].”
Deep Learning on Lie Groups for Skeleton-Based Action Recognition (2016, methodology): “While the convergence of the used SGD algorithm on Riemannian manifolds has been studied well in [8, 6] already, the convergence behavior (see Fig.”
First-order Methods for Geodesically Convex Optimization (2016, methodology): “When specialized to Hadamard manifolds, our result reduces to the analysis in (Bonnabel, 2013), which in turn relies on (Cordero-Erausquin et al., 2001, Lemma 3.12).”
Relative gradient optimization of the Jacobian term in unsupervised deep learning (2020, neutral): “[8]).”
Multi-relational Poincaré Graph Embeddings (2019, methodology): “For fairness of comparison, we optimize the Euclidean model using stochastic gradient descent (SGD) and the hyperbolic model using Riemannian stochastic gradient descent (RSGD) [4].”
Poincaré GloVe: Hyperbolic Word Embeddings (2018, result): “Empirically, we indeed noticed better results using R ADAGRAD , rather than simply Riemannian SGD (Bonnabel, 2013).”
Manopt, a matlab toolbox for optimization on manifolds (2013, background): “More solvers can be added, with an outlook toward Riemannian BFGS (Ring & Wirth, 2012), stochastic gradients (Bonnabel, 2013), nonsmooth subgradients schemes (Dirr et al. , 2007), etc.”
Hyperbolic Neural Networks (2018, methodology): “Thus, they are optimized using full Riemannian stochastic gradient descent (RSGD) [5, 11].”
Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry (2018, methodology): “Following Bonnabel (2013), we minimize Equation (6) using Riemannian SGD.”
A Riemannian Network for SPD Matrix Learning (2016, methodology): “The convergence analysis of the used SGD algorithm on Riemannian manifolds follows the developments in (Bottou 2010; Bonnabel 2013).”
Building Deep Networks on Grassmann Manifolds (2016, methodology): “Furthermore, the convergence of the stochastic gradient descent (SGD) algorithm on Riemannian manifolds has also been studied well in (Bottou 2010; Bonnabel 2013).”
Orthogonal Weight Normalization: Solution to Optimization over Multiple Dependent Stiefel Manifolds in Deep Neural Networks (2017, background): “It can breed further theoretical researches in stochastic optimization over matrix manifolds [4].”
Transfer Learning: A Riemannian Geometry Framework With Applications to Brain–Computer Interfaces (2018, background): “Some of them try to find the right value through numerical procedure like deterministic line search [17], [26], simple or stochastic gradient descent [10], [31].”
Hyperbolic Graph Convolutional Neural Networks (2019, methodology): “Note that to train shallow Poincaré embeddings, we use Riemannian Stochastic Gradient Descent [4, 48], since its model parameters are hyperbolic.”
Riemannian Adaptive Optimization Methods (2018, methodology): “(1)): By analogy, note that the size of an RSGD update in M i (see Eq.”
Hyperbolic Entailment Cones for Learning Hierarchical Embeddings (2018, methodology): “The exponential map is important, for instance, when performing gradient-descent over parameters lying in a manifold (Bonnabel, 2013).”
Poincaré Embeddings for Learning Hierarchical Representations (2017, methodology): “Since the Poincaré Ball has a Riemannian manifold structure, we can optimize Equation (2) via stochastic Riemannian optimization methods such as RSGD [6] or RSVRG [39].”
Machine Learning on Graphs: A Model and Comprehensive Taxonomy (2020, background): “Note that since the hyperbolic space has a manifold structure, embeddings need to be optimized using Riemannian optimization techniques [12].”