Efficient Riemannian Optimization on the Stiefel Manifold via the Cayley Transform

By Jun Li, Fuxin Li, Sinisa Todorovic. 2020

146 citations recorded by the source corpus.

Strictly enforcing orthonormality constraints on parameter matrices has been shown advantageous in deep learning. This amounts to Riemannian optimization on the Stiefel manifold, which, however, is computationally expensive. To address this challenge, we present two main contributions: (1) A new efficient retraction map based on an iterative Cayley transform for optimization updates, and (2) An implicit vector transport mechanism based on the combination of a projection of the momentum and the Cayley transform on the Stiefel manifold. We specify two new optimization algorithms: Cayley SGD with momentum, and Cayley ADAM on the Stiefel manifold. Convergence of Cayley SGD is theoretically analyzed. Our experiments for CNN training demonstrate that both algorithms: (a) Use less running time per iteration relative to existing approaches that enforce orthonormality of CNN parameters; and (b) Achieve faster convergence rates than the baseline SGD and ADAM algorithms without compromising the performance of the CNN. Cayley SGD and Cayley ADAM are also shown to reduce the training time for optimizing the unitary transition matrices in RNNs.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Continual Learning in Low-rank Orthogonal Subspaces (2020, background): “Li et al. [2020] showed that one can bypass the expensive matrix inversion in (6) by following the fixed-point iteration of the Cayley transform, Y (τ) = Wl − τ 2 A(Wl + Y (τ)).”
SpinQuant: LLM quantization with learned rotations (2024, methodology): “As a result, we propose SpinQuant that integrates and optimizes the rotation matrix to minimize the final loss of the quantized network, with fixed weight parameters, by employing the Cayley SGD [21], a proficient technique for optimizing orthonormal matrices.”
Orthogonal Over-Parameterized Training (2020, methodology): “It is essentially conducting gradient descent on the Stiefel manifold [44, 74, 75, 42, 3, 22, 33].”
Unsupervised Energy-based Out-of-distribution Detection using Stiefel-Restricted Kernel Machine (2021, methodology): “For St-RKM, an alternating minimization training scheme is used: the weights of the encoder φθ(·) and of the decoder ψξ(·) are jointly optimized with the Adam optimizer [22] with learning rate 2×10−4, and the interconnection matrix U is optimized with the Cayley Adam optimizer [23] with learning rat”
Normalization Techniques in Training DNNs: Methodology, Analysis and Application (2020, methodology): “Most Riemannian optimization methods in the deep learning community address the second phase by either QR-decomposition-type retraction [164], [165] or Cayley transformation [144], [146], [162], [166].”