Information Theoretical Analysis of Multivariate Correlation

By Michael Satosi Watanabe. 1960

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Theory

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, methodology): “The FactorVAE (Kim and Mnih, 2018) and the β-TCVAE (Chen et al., 2018) penalize the total correlation (Watanabe, 1960) with adversarial training (Nguyen et al., 2010; Sugiyama et al., 2012) or with a tractable but biased Monte-Carlo estimator respectively.”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, methodology): “β-VAE restricts the capacity of the information bottleneck by penalizing the KL-divergence, using β = λ1 > 1 with R1(qφ(z|x)) := DKL[qφ(z|x)||p(z)], and λ2 = 0; FactorVAE penalizes the Total Correlation [77] of the latent variables via adversarial training, using λ1 = 0 and λ2 = 1 with R2(qφ(z)) := ”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, methodology): “The FactorVAE (Kim & Mnih, 2018) and the β-TCVAE (Chen et al., 2018) penalize the total correlation (Watanabe, 1960) with adversarial training (Nguyen et al., 2010; Sugiyama et al., 2012) or with a tractable but biased Monte-Carlo estimator respectively.”
Disentangling by Factorising (2018, background): “KL(q(z)||q̄(z)) is known as Total Correlation (TC, Watanabe, 1960), a popular measure of dependence for multiple random variables.”
Isolating Sources of Disentanglement in VAEs (2018, background): “In information theory, ii is referred to as the total correlation (TC), one of many generalizations of mutual information to more than two random variables [20].”
Information Dropout: Learning Optimal Representations Through Noisy Computation (2016, methodology): “We chose to use Total Correlation both for historical reasons, after its introduction in [19], and to emphasize the relation with disentanglement also in recent work on unsupervised learning [21].”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “To reinforce only the former effect, they propose to regularize L VAE with the total correlation TC ( q φ ( z )) = D KL ( q φ ( z ) j q φ ( z j )) of q φ ( z ) —a popular measure of dependence for multiple random variables [47].”
Learning Discrete and Continuous Factors of Data via Alternating Disentanglement (2019, background): “(2018) then analyzed the KL divergence term under the expectation over the data distribution could be broken down into the mutual information term between the data and the latent code, and the KL divergence term between the latent distribution and the factorial prior often denoted as total correlati”
Auto-Encoding Total Correlation Explanation (2018, methodology): “A measure of multivariate mutual information called total correlation (Watanabe, 1960) or multi-information (Studenỳ & Vejnarova, 1998) is defined as follows: TC (x) = d∑ i=1 H (xi)−H (x) = DKL ( p(x)|| d∏ i=1 p(xi) ) (1) Note that DKL (·) denotes the Kullback-Leibler divergence in Eq.”
A review of feature selection methods based on mutual information (2013, background): “redundancy, such as the Markov blanket [33, 67] and total correlation [62].”
Mutual Information Analysis: a Comprehensive Study (2011, background): “Eventually, it is possible to use the total correlation [36] C(X,Y1, Y2) = H[X] + H[Y1] + H[Y2]−H[X,Y1, Y2] that can also be written as:”
Isolating Sources of Disentanglement in Variational Autoencoders (2018, background): “In information theory, ii is referred to as the total correlation (TC), one of many generalizations of mutual information to more than two random variables (Watanabe, 1960).”
On Measures of Entropy and Information (1961, unknown): “…if we extend the notation to allow colons ’ : ’ in conditionals (i.e. on right side of a bar ’ | ’) [1], e.g. Total correlation (Multi-information, multivariate constraint, redundancy, integration) [17, 20, 37, 58] The total amount of information carried by correlations between the variables.”
Measuring organizational performance and organizational excellence of SMEs – Part 1: a conceptual framework (2010, background): “In probability theory and in particular, information theory, total correlation is one of several generalizations of the mutual information ( Watanabe, 1960”
Information Constraints on Auto-Encoding Variational Bayes (2018, background): “The β-TCVAE [6] penalized an approximation of the total correlation (TC), defined as DKL(q̂φ(z) ∣∣ ∏k q̂φ(zk)) [26], which is a measure of multivariate mutual independence.”
Maximally Informative Hierarchical Representations of High-Dimensional Data (2014, background): “The resulting objective could also be interpreted as using a distortion measure that preserves the most redundant information in a high dimensional system.”
Interaction information for causal inference: The case of directed triangle (2017, background): “The most well known generalizations are total correlation [1] (also known as multi-information [2]), and interaction information [3], [4].”