Contrastive Learning as Goal-Conditioned Reinforcement Learning
By Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, Sergey Levine. Neural Information Processing Systems, 2022
297 citations recorded by the source corpus.
In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e.g., auxiliary losses, data augmentation). How can we design RL algorithms that directly acquire good representations? In this paper, instead of adding representation learning parts to an existing RL algorithm, we show (contrastive) representation learning methods can be cast as RL algorithms in their own right. To do this, we build upon prior work and apply contrastive representation learning to action-labeled trajectories, in such a way that the (inner product of) learned representations exactly corresponds to a goal-conditioned value function. We use this idea to reinterpret a prior RL method as performing contrastive learning, and then use the idea to propose a much simpler method that achieves similar performance. Across a range of goal-conditioned RL tasks, we demonstrate that contrastive RL methods achieve higher success rates than prior non-contrastive methods, including in the offline RL setting. We also show that contrastive RL outperforms prior methods on image-based tasks, without using data augmentation or auxiliary objectives.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/53dcf467fbded741dd08902d4203a9b57e889c87
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Vikash Kumar
- Pieter Abbeel
- Amy Zhang
- Shagun Sodhani
- Dinesh Jayaraman
- Abhishek Gupta
Citations in the corpus
- BridgeData V2: A Dataset for Robot Learning at Scale (2023, methodology): “To validate this, we specifically select a wide variety of methods in our evaluation to show that our work provides a single dataset that is compatible with many approaches [7, 47, 48, 17].”
- Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, background): “Similarly, goal-states have also extended the domain of reinforcement learning through Goal-Conditioned Reinforcement Learning (GCRL) [8, 9, 22, 32], where the agent is not provided expert demonstrations but reward signals instead.”
- VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training (2022, background): “Eysenbach et al. (2022) shows that goal-conditioned Q-learning with a particular choice of reward function can be understood as performing contrastive learning.”