Applying Discrete PCA in Data Analysis

By Wray L. Buntine, Aleks Jakulin. 2004

152 citations recorded by the source corpus.

Methods for analysis of principal components in discrete data have existed for some time under various names such as grade of membership modelling, probabilistic latent semantic analysis, and genotype inference with admixture. In this paper we explore a number of extensions to the common theory, and present some application of these methods to some common statistical tasks. We show that these methods can be interpreted as a discrete version of ICA. We develop a hierarchical version yielding components at different levels of detail, and additional techniques for Gibbs sampling. We compare the algorithms on a text prediction task using support vector machines, and to information retrieval.

Subfields: Artificial Intelligence, Molecular Biology, Statistics and Probability, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Statistics and Probability

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Rethinking LDA: Moment Matching for Discrete ICA (2015, neutral): “This model was introduced by Canny [12] and later named as a discrete ICA model [13].”
Causal Inference through a Witness Protection Program (2014, methodology): “In practice, brute-force can be still valuable by using a method such as discrete PCA [3] to reduceW\{W} to a small set of binary variables.”
Streaming Variational Bayes (2013, background): “…over stochastic variational inference and the sufﬁcient statistics update algorithm, particularly with respect to the key issue of obtaining approximations to posterior probabilities based on the number of documents seen thus far, not posterior probabilities for a ﬁxed number of documents.”
Deep Exponential Families (2014, methodology): “Latent tree hierarchies have been constructed with speciﬁc distributions (Dirchlet) [21], while Bayesian factor analysis methods such as exponential family PCA [24] and multinomial PCA [8] can be seen as a single layer deep exponential family.”
WordICA—emergence of linguistic representations for words by independent component analysis (2010, result): “It would also be interesting to adapt the topical analysis models of latent Dirichlet allocation (Blei, Ng and Jordan 2003) and discrete PCA (Buntine and Jakulin 2004) to word analysis and compare the results to ICA.”
Learnable Explicit Density for Continuous Latent Space and Variational Inference (2017, unknown): “…the introduction of a latent representation requires approximate inference, as the model distribution is biased by simplifying posterior densities (Buntine & Jakulin, 2004); i.e. training is achieved by maximizing the variational lower bound on the marginal log likelihood: where subscripts θ, φ,…”
The rate adapting poisson model for information retrieval and object recognition (2006, background): “As pointed out by Buntine and Jakulin (2004), the basic architecture of LDA is known under various names such as add-mixtures, grade of memberships model, multiple aspect model and multinomial PCA.”
Semi-Supervised Sequence Modeling with Syntactic Topic Models (2005, methodology): “Similar to the approach taken by Buntine (2004) and Miller, Guinness, & Zamanian (2004), we create word clusters from unlabeled data and use them as features for supervised training.”
Supervised Topic Models (2007, background): “The complexity of document corpora has led to considerable interest in applying hierarchical statistical models based on what are called topics [3, 12, 8, 5 ]. Formally, a topic is a probability distribution over terms in a vocabulary.”
Hidden Topic Markov Models (2007, background): “Modeling the observed text as generated from latent aspects or topics is a prominent approach in machine learning studies of texts (e.g., [8] [14] [3] [4] [6]).”
Continuous Time Dynamic Topic Models (2008, background): “Topic models, such as latent Dirichlet allocation (LDA) and the more general discrete component analysis [3, 4], posit that a small number of distributions over words, called topics, can be used to explain the observed collection.”
Automatic Evaluation of Topic Coherence (2010, methodology): “LDA is a Bayesian graphical model for text document collections represented by bags-of-words (see Blei et al. (2003), Grifﬁths and Steyvers (2004), Buntine and Jakulin (2004)).”
Cross-domain collaboration recommendation (2012, unknown): “The new distributionθvv′ is then defined as̃θv + θ̃v′ , a simple mixture of the two expanded multinomials ofθv andθv′ [5].”