Theory and Evaluation Metrics for Learning Disentangled Representations

By Kien Do, Truyen Tran. 2019

109 citations recorded by the source corpus.

We make two theoretical contributions to disentanglement learning by (a) defining precise semantics of disentangled representations, and (b) establishing robust metrics for evaluation. First, we characterize the concept "disentangled representations" used in supervised and unsupervised methods along three dimensions-informativeness, separability and interpretability - which can be expressed and quantified explicitly using information-theoretic constructs. This helps explain the behaviors of several well-known disentanglement learning models. We then propose robust metrics for measuring informativeness, separability and interpretability. Through a comprehensive suite of experiments, we show that our metrics correctly characterize the representations learned by different methods and are consistent with qualitative (visual) results. Thus, the metrics allow disentanglement learning methods to be compared on a fair ground. We also empirically uncovered new interesting properties of VAE-based methods and interpreted them with our formulation. These findings are promising and hopefully will encourage the design of more theoretically driven models for learning disentangled representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lesia Semenova
2. Chudi Zhong
3. Zhi Chen
4. Haiyang Huang
5. Cynthia Rudin
6. Tong Wang
7. Octavio Loyola-González
8. Berk Ustun

In Computer Vision and Pattern Recognition

1. Zhi Chen
2. Lesia Semenova
3. Chudi Zhong
4. Haiyang Huang
5. Cynthia Rudin

In Representation & Self-Supervised Learning

1. Lesia Semenova
2. Chudi Zhong
3. Zhi Chen
4. Haiyang Huang
5. Cynthia Rudin

Citations in the corpus

• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, methodology): “If we use an unsupervised algorithm for disentangling the space, and then evaluate whether it indeed disentangled the known concepts, then we have a useful quantitative evaluation (Higgins et al., 2017; Chen et al., 2018b; Kim and Mnih, 2018; Eastwood and Williams, 2018; Do and Tran, 2019).”