Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition

By Justin Fu, Avi Singh, Dibya Ghosh, Larry Yang, Sergey Levine. 2018

133 citations recorded by the source corpus.

The design of a reward function often poses a major practical challenge to real-world applications of reinforcement learning. Approaches such as inverse reinforcement learning attempt to overcome this challenge, but require expert demonstrations, which can be difficult or expensive to obtain in practice. We propose variational inverse control with events (VICE), which generalizes inverse reinforcement learning methods to cases where full demonstrations are not needed, such as when only samples of desired goal states are available. Our method is grounded in an alternative perspective on control and reinforcement learning, where an agent's goal is to maximize the probability that one or more events will happen at some point in the future, rather than maximizing cumulative rewards. We demonstrate the effectiveness of our methods on continuous control tasks, with a focus on high-dimensional observations like images where rewards are hard or even impossible to specify.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

End-to-End Robotic Reinforcement Learning without Reward Engineering (2019, methodology): “by a reinforcement learning algorithm: the RL algorithm can visit parts of the observation space that the classifier was not trained on, causing it to output incorrect probabilities [13].”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Some prior RL methods have also used contrastive learning to acquire reward functions [14, 20, 33, 38, 63, 67, 92, 133, 134, 146], often in imitation learning settings [37, 55].”
Transformers for One-Shot Visual Imitation (2020, background): “There are myriad ways to introduce goal conditioning, including with the robot’s state [24], ”goal” images of the final state [25, 26, 27], natural language [28], and video or images of humans [29, 30].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, methodology): “Similar to [21], [30], further training Cs on this data improves its accuracy by combating the false positive problem and offsetting the”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, background): “While the approach of measuring similarity between agent’s observation and goal has been explored in prior work, for instance by using a learned classiﬁer (Pinto and Gupta, 2016; Fu et al., 2018; Eysenbach et al., 2021), they have all required goal images from the same scene as the agent.”
Few-Shot Goal Inference for Visuomotor Learning and Planning (2018, background): “[30]) is an interesting direction for future work.”
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021, background): “One simple yet effective direction in reward learning is to train a classifier that recognizes task success and use it as basis for a reward function (Pinto & Gupta, 2016; Levine et al., 2018; Fu et al., 2018; Xie et al., 2018).”
SOLAR: Deep Structured Representations for Model-Based Reinforcement Learning (2018, unknown): “This setup is reminiscent of Fu et al. (2018), though our goal is not to classify expert data from policy data.”
Entity Abstraction in Visual Model-Based Reinforcement Learning (2019, background): “Building good cost functions for real world tasks is generally difficult [57] because the underlying state of the environment is always unobserved and can only be modeled through modeling observations.”
The Ingredients of Real-World Robotic Reinforcement Learning (2020, methodology): “To design a system that requires minimal human engineering for providing reward, we use a data-driven reward speciﬁcation framework called variational inverse control with events (VICE) introduced by Fu et al. (2018).”
Zero-Shot Reward Specification via Grounded Natural Language (2022, methodology): “6 we observe that our Full Reward Model performs as well as oracle reward and VICE both of which are priviledged on SawyerSimRobot-Pushing (pushing a blue puck close to a yellow puck), FetchSimRobot-Stacking (stacking a yellow block on top of a red block), and FetchSimRobot-Placing (placing a yellow”