Value Prediction Network

By Junhyuk Oh, Satinder Singh, Honglak Lee. 2017

352 citations recorded by the source corpus.

This paper proposes a novel deep reinforcement learning (RL) architecture, called Value Prediction Network (VPN), which integrates model-free and model-based RL methods into a single neural network. In contrast to typical model-based RL methods, VPN learns a dynamics model whose abstract states are trained to make option-conditional predictions of future values (discounted sum of rewards) rather than of future observations. Our experimental results show that VPN has several advantages over both model-free and model-based baselines in a stochastic environment where careful planning is required but building an accurate observation-prediction model is difficult. Furthermore, VPN outperforms Deep Q-Network (DQN) on several Atari games even with short-lookahead planning, demonstrating its potential as a new way of learning a good state representation.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “[28] Junhyuk Oh, Satinder Singh, and Honglak Lee.”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, methodology): “…can be viewed as a formalization of recent works which use neural networks to learn latent space models of the environment (Ha & Schmidhuber, 2018; Oh et al., 2017; Hafner et al., 2018; Francois-Lavet et al., 2018), because the value functions in the DeepMDP are guaranteed to be good…”
Model-Based Reinforcement Learning for Atari (2019, methodology): “Our method achieves around 330% of the Dyna-DQN score on Asterix, 120% on Q-Bert, 150% on Seaquest and 80% on Ms. Pac-Man.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “Further work employs transition models in order to perform rollouts in the encoded state representation (Oh et al., 2017), and demonstrate modest gains on Atari games (compared to DQN).”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, methodology): “Value prediction networks (VPNs) (Oh et al., 2017) take a similar approach but are more closely related to our work because the learned model components are used in a tree for planning.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “A number of such methods have been explored in the conventional online RL setting (Tamar et al., 2016; Dosovitskiy and Koltun, 2016; Oh et al., 2017), and in the offline RL setting, such techniques have been applied effectively to learning navigational policies for mobile robots from previously coll”
Causally Correct Partial Models for Reinforcement Learning (2020, methodology): “The resulting model may be used in various ways, e.g. for planning (Oh et al., 2017; Schrittwieser et al., 2019), generation of synthetic training data (Weber et al., 2017), better credit assignment (Heess et al., 2015), learning useful internal representations and belief states (Gregor et al.,…”
Self-Supervised Deep Reinforcement Learning with Generalized Computation Graphs for Robot Navigation (2017, background): “Our empirical results show that the design presented in prior work [22], [23] is often not the best one for real-world continuous learning tasks, and shed light on the tradeoffs involved with single- and multi- action Q-learning, as well as purely prediction-based control 1.”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Prior methods have also explored incorporating computation that resembles model-based planning but without constraining the intermediate predictions of the planner to match plausible environment observations (Tamar et al., 2016; Racanière et al., 2017; Oh et al., 2017; Silver et al., 2017).”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, methodology): “Value prediction network (VPN) Taken from Oh et al. (2017), this objective uses an RNN starting at φ ( s t ) and inputting a t : t + k for k steps to predict the k -step future rewards and value functions.”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “[45] all present neural networks whose architectures mimic classical iterative planning algorithms, and which are trained by reinforcement learning or to predict user-defined, high-level features; in these, there is no explicit environment model.”
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning (2018, methodology): “A related approach is proposed by value prediction networks (VPN), which expand encoded state predictions and perform backups on a set of expanded imagined paths to compute an improved target estimate (Oh et al., 2017).”
State Representation Learning for Control: An Overview (2018, methodology): “In these games, states can be represented through diﬀerent variables (time in achieving a task, amount of bonus, keeping alive, etc.) [Shelhamer et al., 2017, Oh et al., 2017].”
Dream to Control: Learning Behaviors by Latent Imagination (2019, background): “When the sensory inputs are high-dimensional images, latent dynamics models can abstract observations to predict forward in compact state spaces (Watter et al., 2015; Oh et al., 2017; Gregor et al., 2019).”
Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation (2018, background): “fact, we seek the help of recent advance of model-based RL [6,7] for this task.”
Learning Plannable Representations with Causal InfoGAN (2018, background): “Recent work in deep RL explored learning goal-conditioned value functions and policies [2, 32], and policies with an explicit planning computation [43, 31, 39].”