Tensor decompositions for learning latent variable models

By Anima Anandkumar, Rong Ge, Daniel J. Hsu, Sham M. Kakade, Matus Telgarsky. 2012

1,214 citations recorded by the source corpus.

This work considers a computationally and statistically efficient parameter estimation method for a wide class of latent variable models--including Gaussian mixture models, hidden Markov models, and latent Dirichlet allocation--which exploits a certain tensor structure in their low-order observable moments (typically, of second- and third-order). Specifically, parameter estimation is reduced to the problem of extracting a certain (orthogonal) decomposition of a symmetric tensor derived from the moments; this decomposition can be viewed as a natural generalization of the singular value decomposition for matrices. Although tensor decompositions are generally intractable to compute, the decomposition of these specially structured tensors can be efficiently obtained by a variety of approaches, including power iterations and maximization approaches (similar to the case of matrices). A detailed analysis of a robust tensor power method is provided, establishing an analogue of Wedin's perturbation theorem for the singular vectors of matrices. This implies a robust and computationally tractable estimation approach for several popular latent variable models.

Subfields: Artificial Intelligence, Computational Mathematics, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Sergey Levine
  3. Percy Liang
  4. Mohit Bansal
  5. Ruslan Salakhutdinov
  6. Quoc V. Le
  7. Jianfeng Gao
  8. Luke S. Zettlemoyer

In Computational Mathematics

  1. Yuanzhi Li
  2. Rong Ge
  3. Anima Anandkumar
  4. Zehao Dou
  5. Kfir Yehuda Levy
  6. Sham M. Kakade
  7. David Steurer
  8. Michael I. Jordan

In Representation & Self-Supervised Learning

  1. Bernhard Schölkopf
  2. Le Song
  3. Kenji Fukumizu
  4. Alex Smola
  5. Arthur Gretton
  6. Yoshua Bengio
  7. Michael I. Jordan
  8. Francis R. Bach

In Signal Processing

  1. Bernhard Schölkopf
  2. Alex Smola
  3. Arthur Gretton
  4. Francis R. Bach
  5. Le Song
  6. Anima Anandkumar
  7. Yuanzhi Li
  8. Pravesh Kothari

Citations in the corpus