Disentangled Representations in Neural Models

By William F. Whitney. 2016

18 citations recorded by the source corpus.

Representation learning is the foundation for the recent success of neural network models. However, the distributed representations generated by neural networks are far from ideal. Due to their highly entangled nature, they are di cult to reuse and interpret, and they do a poor job of capturing the sparsity which is present in real- world transformations. In this paper, I describe methods for learning disentangled representations in the two domains of graphics and computation. These methods allow neural methods to learn representations which are easy to interpret and reuse, yet they incur little or no penalty to performance. In the Graphics section, I demonstrate the ability of these methods to infer the generating parameters of images and rerender those images under novel conditions. In the Computation section, I describe a model which is able to factorize a multitask learning problem into subtasks and which experiences no catastrophic forgetting. Together these techniques provide the tools to design a wide range of models that learn disentangled representations and better model the factors of variation in the real world.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “…are disentangled can be approached by optimizing for some notion of (statistical) independence between features (e.g. Schmidhuber, 1992c; Chen et al., 2016; Higgins et al., 2017a), sparse feature updates across time (Whitney, 2016), or independent controllability of features (Thomas et al., 2017).”
Understanding Visual Concepts with Continuation Learning (2016, methodology): “To learn the gating function, we use a technique ﬁrst described in (W. Whitney 2016) for smoothly annealing a soft weighting function into a binary decision.”
SE3-nets: Learning rigid body motion using deep neural networks (2016, methodology): “To bias the network to model rigid motion, we adapt a weight sharpening technique used in [7] and show that this results in a segmentation of the environment into distinct objects along with their rigid motion.”
A Framework for the Quantitative Evaluation of Disentangled Representations (2018, background): “The same cannot be said for tasks in a nonhuman domain like stock prediction, where codes that make sense to us diverge from those that best represent the underlying factors of variation (Whitney, 2016).”