Identifiability of parameters in latent structure models with many observed variables

By Elizabeth S. Allman, Catherine Matias, John A. Rhodes. 2008

589 citations recorded by the source corpus.

While hidden class models of various types arise in many statistical applications, it is often difficult to establish the identifiability of their parameters. Focusing on models in which there is some structure of independence of some of the observed variables conditioned on hidden ones, we demonstrate a general approach for establishing identifiability utilizing algebraic arguments. A theorem of J. Kruskal for a simple latent-class model with finite state space lies at the core of our results, though we apply it to a diverse set of models. These include mixtures of both finite and nonparametric product distributions, hidden Markov models and random graph mixture models, and lead to a number of new results and improvements to old ones. In the parametric setting, this approach indicates that for such models, the classical definition of identifiability is typically too strong. Instead generic identifiability holds, which implies that the set of nonidentifiable parameters has measure zero, so that parameter inference is still meaningful. In particular, this sheds light on the properties of finite mixtures of Bernoulli products, which have been used for decades despite being known to have nonidentifiable parameters. In the nonparametric setting, we again obtain identifiability only when certain restrictions are placed on the distributions that are mixed, but we explicitly describe the conditions.

Subfields: Artificial Intelligence, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Causal Effect Inference with Deep Latent-Variable Models (2017, neutral): “For example, if X includes three independent views of Z [4, 22, 16, 2]; if Z is categorical and X is a Gaussian mixture model with components determined by X [5]; or if Z is comprised of binary variables and X are so-called “noisy-or” functions of Z [23, 8].”
Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Series (2020, methodology): “This observation raises the question of identiﬁability since mixture models with non-parametric emission distributions are generally unidentiﬁable, though many commonly used parametric forms are identiﬁable (Allman et al., 2009).”
Deconfounding Reinforcement Learning in Observational Settings (2018, methodology): “Despite this, identification is possible in both our model and the one described by Louizos et al. (2017) under specific conditions (Louizos et al., 2017; Kuroki and Pearl, 2014; Miao et al., 2018; Pearl, 2012; Allman et al., 2009).”
Identifying Finite Mixtures of Nonparametric Product Distributions and Causal Inference of Confounders (2013, methodology): “, that {P (Xj |z)}i≤m is a linearly independent set, is also used by Allman et al. (2009) (see Sec.”
Tensor decompositions for learning latent variable models (2012, background): “In these contexts, it is often also possible to establish strong identifiability results, without giving an explicit estimators, by invoking the non-constructive identifiability argument of Kruskal (1977)—see the article by Allman et al. (2009) for several examples.”
Exponentially vanishing sub-optimal local minima in multilayer neural networks (2017, background): “In (Soudry & Carmon, 2016), it was noticed that an inﬁnitesimal perturbation of A makes the matrix A ◦ X full rank with probability 1 (Allman et al., 2009, Lemma 13) – which entails that e = 0 at all DLMs.”
A concise proof of Kruskal’s theorem on tensor decomposition (2009, methodology): “Recently, Kruskal’s theorem has been used as a general tool for investigating the identifiability of a wide variety of statistical models with hidden variables [1, 2].”
Non‐parametric estimation of finite mixtures from repeated measurements (2016, methodology): “These approaches are applicable more generally than are the earlier proposals of Hettmansperger and Thomas (2000) and Elmore et al. (2004), and of Hall and Zhou (2003) and Hall et al. (2005).”
Statistical clustering of temporal networks through a dynamic stochastic block model (2015, methodology): “We refer to Allman et al. (2009, 2011) for more details.”
FIXED POINTS OF THE EM ALGORITHM AND NONNEGATIVE RANK BOUNDARIES (2013, methodology): “Such models include phylogenetic models [2, 4] and hidden Markov models [12].”
Consistency of maximum-likelihood and variational estimators in the Stochastic Block Model (2011, methodology): “, 2011), or on moment estimation in a particular instance of SBM called affiliation model (Ambroise and Matias, 2012, and also Example 1 in the present paper). A variational approach has been proposed by Daudin et al. (2008) to remedy this computational burden. It can be used with binary directed SB”
Latent Variable Models (2019, background): “Under 4 latent classes, with 4 raters and each classifies the tumor biopsies into 3 classes, the model estimates are identifiable up to a permutation of the underlying truth (Allman et al., 2009).”