Value Iteration Networks

By Aviv Tamar, Sergey Levine, Pieter Abbeel, Yi Wu, Garrett Thomas. 2016

697 citations recorded by the source corpus.

We introduce the value iteration network (VIN): a fully differentiable neural network with a `planning module' embedded within. VINs can learn to plan, and are suitable for predicting outcomes that involve planning-based reasoning, such as policies for reinforcement learning. Key to our approach is a novel differentiable approximation of the value-iteration algorithm, which can be represented as a convolutional neural network, and trained end-to-end using standard backpropagation. We evaluate VIN based policies on discrete and continuous path-planning domains, and on a natural-language based search task. We show that by learning an explicit planning computation, VIN policies generalize better to new, unseen domains.

Subfields: Artificial Intelligence, Reinforcement Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Model Architectures

Citations in the corpus

Learning and Querying Fast Generative Models for Reinforcement Learning (2018, background): “Tamar et al. (2016) propose an architecture that is designed to learn the value-iteration algorithm which requires knowledge about environment transitions.”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, background): “In deep RL, value iteration networks (Tamar et al., 2016) use a learned differentiable model to plan on the ﬂy, but require planning over the full state space, which must also possess a spatial structure with local dynamics such that convolution operations can execute the planning algorithm.”
Value Prediction Network (2017, neutral): “Our VPNs are related to Value Iteration Networks [35] (VINs) which perform value iteration (VI) by approximating the Bellman-update through a convolutional neural network (CNN).”
The Predictron: End-To-End Learning and Planning (2016, background): “Value iteration networks (VINs) (Tamar et al., 2016) also learn value functions end-to-end using an internal model, similar to the (non-λ) predictron.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “A number of such methods have been explored in the conventional online RL setting (Tamar et al., 2016; Dosovitskiy and Koltun, 2016; Oh et al., 2017), and in the offline RL setting, such techniques have been applied effectively to learning navigational policies for mobile robots from previously coll”
Learning to reinforcement learn (2016, background): “In particular, Ilin et al. (2007) showed that recurrent networks trained on random mazes can approximate dynamic programming procedures (see also Silver et al., 2017; Tamar et al., 2016).”
A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, positive): “Planning is done on this egocentric map utilizing Value Iteration Networks (VIN) (Tamar et al., 2016).”
Deriving Neural Architectures from Sequence and Graph Kernels (2017, background): “For example, value iteration calculations can be folded into convolutional architectures so as to optimize the representations to facilitate planning (Tamar et al., 2016).”
The limits and potentials of deep learning for robotics (2018, background): “…re-composition of component solutions from different places on the spectrum can be achieved with differentiable versions of existing algorithms (one end of the spectrum) that are compatible solutions obtained with back-propagation (other end of the spectrum) [16], [39], [50], [108], [116], [128].”
MOPO: Model-based Offline Policy Optimization (2020, methodology): “While prior approaches have used these models to select actions using planning [66, 16, 53, 50, 58], we choose to build upon Dyna-style approaches that optimize for a policy [63, 65, 71, 31, 25, 27, 43], speciﬁcally MBPO [28].”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Prior methods have also explored incorporating computation that resembles model-based planning but without constraining the intermediate predictions of the planner to match plausible environment observations (Tamar et al., 2016; Racanière et al., 2017; Oh et al., 2017; Silver et al., 2017).”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, background): “Several prior works (Tamar et al., 2016; Amos et al., 2018; Srinivas et al., 2018) have used imitation learning to train policies that contain planning-like modules as part of the model architecture.”
Imagination-Augmented Agents for Deep Reinforcement Learning (2017, background): “Tamar et al. [35], Silver et al. [36], and Oh et al. [37] all present neural networks whose architectures mimic classical iterative planning algorithms, and which are trained by reinforcement learning or to predict user-deﬁned, high-level features; in these, there is no explicit environment model.”
Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics (2017, neutral): “Progress in model-based Deep RL has thus far been limited, though methods like Embed to Control (Watter et al., 2015), Value Iteration Networks (Tamar et al., 2016), and the Predictron (Silver et al., 2016b) demonstrate the promise of this direction.”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “In context of navigation, learning and DRL has been used to obtain policies [4,12,13,21,35,50,56,59,64].”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Generalization : There is a recent trend in reinforcement learning focusing on the problem of generalization, ranging from learning to plan (Tamar et al., 2016), meta-learning (Duan et al., 2016; Finn et al., 2017a) to zero-shot learning (Andreas et al., 2016; Oh et al., 2017; Higgins et al., 2017).”
A Brief Survey of Deep Reinforcement Learning (2017, neutral): “search [108] and planning [118], a deeper integration with other traditional AI approaches promises benefits such as better sample complexity, generalisation and interpretability [30].”
Learning model-based planning from scratch (2017, background): “The value iteration network [22] and predictron [20] both train deep networks to implicitly plan via iterative rollouts.”
Semi-parametric Topological Memory for Navigation (2018, methodology): “Gupta et al. (2017) build a 2D multi-scale metric map using the end-to-end trainable planning approach of Tamar et al. (2016).”