Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation

By Pierre Sermanet, Corey Lynch, Jasmine Hsu, Sergey Levine. 2017

162 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Representation Learning with Contrastive Predictive Coding (2018, background): “More recent work includes Time Contrastive Networks [24] which proposes to minimize distances between embeddings from multiple viewpoints of the same scene and whilst maximizing distances between embeddings extracted from different timesteps.”
Unsupervised state representation learning with robotic priors: a robustness benchmark (2017, methodology): “We therefore use an assessment of the representation’s quality which is based on a Nearest-Neighbors approach as in [19] for example.”
Temporal Relational Reasoning in Videos (2017, methodology): “Time contrast networks are used for self-supervised imitation learning of object manipulation from third-person video observation [26].”
Playing hard exploration games by watching YouTube (2018, methodology): “In ﬁgure 11, we visualize the t-SNE projection of four observation sequences using embeddings computed using TCN, TDC, L 3 -Net, CMC and our combined method.”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “a reward function based on human demonstrations [47, 49, 26, 46, 50].”
State Representation Learning for Control: An Overview (2018, methodology): “For example it can be used to learn similarity metrics or time dependencies, as it is done in time-contrastive networks [Sermanet et al., 2017].”
Imitation Learning from Video by Leveraging Proprioception (2019, methodology): “…other imitation from observation algorithms that do not exploit the imitator’s proprioceptive state information: Time Contrastive Networks (TCN) [ Sermanet et al. , 2018 ] , Behavioral Cloning from Observation (BCO) [Torabi et al. , 2018], Generative Adversarial Imitation Learning (GAIfO)…”
PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “Many recent works have trained agents such as simulated and real robots from the raw pixels of videos of humans [13, 37, 38, 48].”
Generative Adversarial Imitation from Observation (2018, background): “Most IfO work thus far (Liu et al., 2017; Sermanet et al., 2017) has focused on perception and not on control.”
S-RL Toolbox: Environments, Datasets and Evaluation Metrics for State Representation Learning (2018, methodology): “We use an assessment of the representation’s quality based on a Nearest-Neighbours approach (as in (Sermanet et al., 2017)).”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “Contrastive learning has also been used in recent works to discover a meaningful latent space and extract reward signals for RL (Sermanet et al., 2017; Warde-Farley et al., 2018; Dwibedi et al., 2018; Laskin et al., 2020).”
Exploiting Hierarchy for Learning and Transfer in KL-regularized RL (2019, background): “Works that transfer knowledge across different bodies include (Devin et al., 2017; Gupta et al., 2017; Sermanet et al., 2017; Xie et al., 2018).”
CLOCS: Contrastive Learning of Cardiac Signals (2020, background): “Bachman et al. (2019) Time Contrastive Networks (Sermanet et al., 2017) attempt to learn commonalities across views and differences across time.”
Unsupervised Visuomotor Control through Distributional Planning Networks (2019, background): “Prior works have aimed to learn representations for control through auto-encoding [27, 48, 15, 16, 34], pretrained supervised features [40], spatial structure [15, 16, 23], and viewpoint invariance [41].”
Reinforcement Learning Upside Down: Don't Predict Rewards - Just Map Them to Actions (2019, methodology): “In particular, we can apply it in an elegant and straighforward way to train robots on learning by demonstration tasks [78, 41, 2, 8, 58] considered notoriously difﬁcult in traditional robotics.”
Imitating Latent Policies from Observation (2018, background): “Similarly, time-contrastive networks (Sermanet et al., 2017) and unsupervised perceptual rewards (Sermanet et al.”
Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation (2017, methodology): “Training an end-to-end model from scratch for each task may be inefficient in practice, and combining our method with higher level representations proposed in prior work would likely lead to more efficient training (Sermanet et al., 2017).”
Universal Planning Networks (2018, methodology): “Value iteration networks of Tamar et al. (2016) embed an approximate differentiable value iteration computation, though their architecture only supports discrete planning and is evaluated on tasks with sparse state transition probabilities. We seek a more general planning computation for more comple”