Integrated Architectures for Learning, Planning, and Reacting Based on Approximating Dynamic Programming

By Richard S. Sutton. 1990

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

What can the brain teach us about building artificial intelligence? (2016, neutral): “Similarly to how probabilistic computations can be amortized for efficiency (see previous section), plans can be amortized into cached values by allowing the model-based system to simulate training data for the model-free system (Sutton, 1990).”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, neutral): “ should be taken less often, and the stochastic exploration strategy has the form of a Boltzmann-like distribution, with the Q-function acting as the negative energy. A large number of prior methods (Sutton, 1990; Kaelbling et al., 1996) have proposed to use such a policy distribution as an explorat”
Decision Transformer: Reinforcement Learning via Sequence Modeling (2021, neutral): “damentally model-free in nature as well. Furthermore, TD-learning is the dominant paradigm in RL for sample efﬁciency, and also features prominently as a sub-routine in many model-based RL algorithms [16, 17]. We also compare with behavior cloning and variants, since it also involves a likelihood ba”
When to use parametric models in reinforcement learning? (2019, positive): “For instance, in Dyna [Sutton, 1990], learning and planning are combined by using new experience to learn both the model and the agent’s predictions, and then planning to further improve the predictions.”
TreeQN and ATreeC: Differentiable Tree Planning for Deep Reinforcement Learning (2017, neutral): “An early example is Dyna-Q (Sutton, 1990) which trains a model-free algorithm with samples drawn from a learned model.”
Value Prediction Network (2017, neutral): “Dyna-Q [32, 34, 39] integrates model-free and modelbased RL by learning an observation-prediction model and using it to generate samples for Q-learning in addition to the model-free samples obtained by acting in the real environment.”
The Predictron: End-To-End Learning and Planning (2016, neutral): “In model-based reinforcement learning architectures such as Dyna (Sutton, 1990), value functions may be updated using both real and imagined trajectories.”
Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, neutral): “This is most obvious in model-based RL, where it is used to train predictive models that can then be used for planning or policy learning (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015b; Janner et al., 2020; Amos et…”
Deep Reinforcement Learning with Double Q-Learning (2015, neutral): “ormation about which states are more valuable than others, directly affecting the quality of the learned policies. The overestimations should not be confused with optimism in the face of uncertainty (Sutton, 1990; Agrawal, 1995; Kaelbling et al., 1996; Auer et al., 2002; Brafman and Tennenholtz, 200”
Neural Approaches to Conversational AI (2018, neutral): “We review several recent proposals, including the ones based on Bayesian models, curiosity-driven strategy, hierarchical reinforcement learning, adversarial learning, and the Dyna framework (Sutton, 1990; Peng et al., 2018) to integrate planning and learning, etc.”
Continuous Deep Q-Learning with Model-based Acceleration (2016, neutral): “…it will be useful to also deﬁne the value function V π ( x t , u t ) and advantage function A π ( x t , u t ) of a given policy π : (Deisenroth et al., 2013; Kober & Peters, 2012), two are particularly relevant for this work: iterative LQG (iLQG) (Li & Todorov, 2004) and Dyna-Q (Sutton, 1990).”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, neutral): “Prior work has explored model-based (Sutton, 1990; Heess et al., 2015; Gu et al., 2016) and off-policy critic-based gradient estimators (Lillicrap et al., 2015).”
Deep Reinforcement Learning for Autonomous Driving: A Survey (2020, neutral): “nt dynamics may reduce the amount of interactions required with the real environment. Moreover, exploration can be performed on the learned 6 models. In the case of model-based approaches (e.g. DynaQ [32], R-max [33]), agents attempt to learn the transition function T and reward function R, which ca”
Reinforcement learning in robotics: A survey (2013, positive): “The idea of combining learning in simulation and in the real environment was popularized by the Dyna-architecture (Sutton, 1990), prioritized sweeping (Moore and Atkeson, 1993), and incremental multi-step Q-learning (Peng and Williams, 1996) in reinforcement learning.”
Meta-Learning Representations for Continual Learning (2019, neutral): “…Riemer et al., 2019; Rebufﬁ et al., 2017; Lopez-Paz and Ranzato, 2017; Aljundi et al., 2019), by using a generative model learned on previous data (Sutton, 1990; Shin et al., 2017), or using knowledge distillation which generates targets using Representation Learning Network (RLN) x 1 x 2 x n .…”
When to Trust Your Model: Model-Based Policy Optimization (2019, neutral): “This branched rollout structure resembles the scheme proposed in the original Dyna algorithm (Sutton, 1990), which can be viewed as a special case of a length 1 branched rollouts.”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, neutral): “…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
Benchmarking Model-Based Reinforcement Learning (2019, neutral): “In the Dyna algorithm [45, 46, 47], training iterates between two steps.”