Understanding Visual Concepts with Continuation Learning

By William F. Whitney, Michael Chang, Tejas D. Kulkarni, Josh Tenenbaum. 2016

55 citations recorded by the source corpus.

We introduce a neural network architecture and a learning algorithm to produce factorized symbolic representations. We propose to learn these concepts by observing consecutive frames, letting all the components of the hidden representation except a small discrete set (gating units) be predicted from the previous frame, and let the factors of variation in the next frame be represented entirely by these discrete gated units (corresponding to symbolic representations). We demonstrate the efficacy of our approach on datasets of faces undergoing 3D transformations and Atari 2600 games.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets (2016, background): “[8] proposed to alleviate the grouping requirement by learning from consecutive frames of images and use temporal continuity as supervisory signal.”
A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
Time-Contrastive Networks: Self-Supervised Learning from Video (2017, background): “Other works have proposed to use prediction as a learning signal [34, 35].”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “For example, a number of works rely on known relations between the factors of variation (Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hs”
Unsupervised Learning of Object Keypoints for Perception and Control (2019, background): “Deep generative models with structured bottlenecks have recently seen a lot of advances [3, 19, 36, 38, 12] but they do not explicitly reason about geometry.”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, neutral): “Due to the initial poor performance of purely unsupervised methods, the ﬁeld initially focused on semi-supervised [60, 11, 55, 56, 43, 45] and weakly supervised approaches [31, 12, 40, 21, 75, 20, 15, 35, 77, 52, 46, 62, 8].”
Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations (2018, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018) or explicit inductive biases (Locatello et al.”
Flexible Neural Representation for Physics Prediction (2018, methodology): “More similar to our approach are inverse graphics methods that extract a lower dimensional physical representation from images that is used to predict physics [25, 26, 51, 50, 52, 53, 7, 49].”
A Compositional Object-Based Approach to Learning Physical Dynamics (2016, background): “…strength, but automatically inferring from visual data what physical laws and object properties are present requires more work in inverse graphics (Chen et al., 2016; Kulkarni et al., 2014; 2015a;b; Whitney et al., 2016) and physics-based visual understanding (Brand, 1997; Wu et al., 2015; 2016).”
Towards a Definition of Disentangled Representations (2018, background): “However, many other approaches [50, 23, 66, 63, 80, 78, 36, 17, 77, 46] and a recent disentanglement metric by [65] consider it perfectly ﬁne for each data generative factor to be encoded by multiple latent dimensions.”
Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “There has been recent work [8, 11, 25, 18, 41, 12, 16] in using deep generative models to disentangle multiple factors of variations (objects, pose, location, etc) from pixel data.”
Variational Inference of Disentangled Latent Concepts from Unlabeled Observations (2017, background): “…about the changes in the generative factors across observations (e.g., sparse changes in consecutive frames of a Video) (Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017), (iv) knowledge of a complementary signal to infer…”
Understanding disentangling in β-VAE (2018, background): “Most initial attempts to learn disentangled representations required supervised knowledge of the data generative factors [18, 35, 32, 44, 43, 12, 24, 7, 42, 20].”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…al., 2012; Cohen & Welling, 2014; 2015; Kulkarni et al., 2015; Hinton et al., 2011; Rippel & Adams, 2013; Reed et al., 2014; Yang et al., 2015; Goroshin et al., 2015; Kulkarni et al., 2015; Cheung et al., 2015; Whitney et al., 2016; Karaletsos et al., 2016; Chen et al., 2016; Higgins et al., 2017).”
Unsupervised Learning of Disentangled Representations from Video (2017, methodology): “[37] use a gating principle to encourage each dimension of the latent representation to capture a distinct mode of variation.”
Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning (2016, background): “Many have suggested that temporal experience with objects as they move and undergo transformations can serve as an important signal for learning about the structure of objects [1, 10, 13, 15, 26, 28, 30, 32, 33, 38, 46, 48, 55, 57, 58].”
Automatically Composing Representation Transformations as a Means for Generalization (2018, background): “Just as the motivation behind disentangled representations [11, 17, 49, 82, 85] is to uncover the latent factors of variation, the motivation behind disentangled programs is to uncover the latent organization of a task.”
Disentangling Factors of Variation Using Few Labels (2019, background): “Exploiting relational information or knowledge of the effect of the factors of variation have both been qualitatively studied to learn disentangled representations [23, 9, 30, 18, 63, 16, 13, 25, 65, 43, 35, 55, 4].”