Riemannian Adaptive Optimization Methods

By Gary Bécigneul, Octavian-Eugen Ganea. International Conference on Learning Representations, 2018

355 citations recorded by the source corpus.

Several first order stochastic optimization methods commonly used in the Euclidean domain such as stochastic gradient descent (SGD), accelerated gradient descent or variance reduced methods have already been adapted to certain Riemannian settings. However, some of the most popular of these optimization tools - namely Adam , Adagrad and the more recent Amsgrad - remain to be generalized to Riemannian manifolds. We discuss the difficulty of generalizing such adaptive schemes to the most agnostic Riemannian setting, and then provide algorithms and convergence proofs for geodesically convex objectives in the particular case of a product of Riemannian manifolds, in which adaptivity is implemented across manifolds in the cartesian product. Our generalization is tight in the sense that choosing the Euclidean space as Riemannian manifold yields the same algorithms and regret bounds as those that were already known for the standard algorithms. Experimentally, we show faster convergence and to a lower train loss value for Riemannian adaptive methods over their corresponding baselines on the realistic task of embedding the WordNet taxonomy in the Poincare ball.

Subfields: Artificial Intelligence, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1f992463673ebd65840fe8d763da98dc94e04c17

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform (2020, methodology): “Becigneul & Ganea (2019) introduce a more general Riemannian optimization, but do not show how to efficiently perform this optimization on the Stiefel manifold.”
Multi-relational Poincaré Graph Embeddings (2019, methodology): “We note that the Riemannian equivalent of adaptive optimization methods has recently been developed [3], but leave replacing SGD and RSGD with their adaptive equivalent to future work.”
Hyperbolic Image Embeddings (2019, methodology): “in the aforementioned layers are naturally elements of Dc n. While in principle it is possible to apply Riemannian optimization techniques to them (e.g., previously proposed Riemannian Adam optimizer [2]), we did not observe any signiﬁcant improvement. Instead, we parametrized them via ordinaryEucli”
Machine Learning on Graphs: A Model and Comprehensive Taxonomy (2020, background): “, 2018; Yu and De Sa, 2019) and challenging Riemannian optimization (Bonnabel, 2013; Becigneul and Ganea, 2018).”
Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group (2019, methodology): “On the other hand, the problem of adapting popular optimization algorithms like RMSProp (Tieleman & Hinton, 2012), ADAM (Kingma & Ba, 2014) or AdaGrad (Duchi et al., 2011) is a topic of current research (Kumar Roy et al., 2018; Becigneul & Ganea, 2019).”
Hyperbolic Graph Neural Networks (2019, methodology): “We use RAMSGrad [4] and AMSGrad for hyperbolic parameters and Euclidean parameters, respectively.”
Poincaré GloVe: Hyperbolic Word Embeddings (2018, methodology): “Vanilla models were optimized with Adagrad (Duchi et al., 2011) while Poincar´e models were optimized with R ADAGRAD (B´ecigneul & Ganea, 2018).”