Dual Policy Iteration

By Wen Sun, Geoffrey J. Gordon, Byron Boots, J. Bagnell. 2018

61 citations recorded by the source corpus.

Recently, a novel class of Approximate Policy Iteration (API) algorithms have demonstrated impressive practical performance (e.g., ExIt from [1], AlphaGo-Zero from [2]). This new family of algorithms maintains, and alternately optimizes, two policies: a fast, reactive policy (e.g., a deep neural network) deployed at test time, and a slow, non-reactive policy (e.g., Tree Search), that can plan multiple steps ahead. The reactive policy is updated under supervision from the non-reactive policy, while the non-reactive policy is improved with guidance from the reactive policy. In this work we study this Dual Policy Iteration (DPI) strategy in an alternating optimization framework and provide a convergence analysis that extends existing API theory. We also develop a special instance of this framework which reduces the update of non-reactive policies to model-based optimal control using learned local models, and provides a theoretically sound way of unifying model-free and model-based RL approaches with unknown dynamics. We demonstrate the efficacy of our approach on various continuous control Markov Decision Processes.

Subfields: Artificial Intelligence, Computer Networks and Communications, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Theoretical analysis of model-based policy learning can provide bounds on the error incurred from the distributional shift due to the divergence between the learned policy π(a|s) and the behavior policy πβ(a|s) (Sun et al., 2018b; Luo et al., 2018; Janner et al., 2019).”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “Although there has been recent interest in monotonic improvement of model-based reinforcement learning algorithms (Sun et al., 2018; Luo et al., 2019), most commonly used model-based approaches lack the improvement guarantees that underpin many model-free methods (Schulman et al., 2015).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “…et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al., 2018).”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, neutral): “We refer to [Kurutach et al., 2018, Clavera et al., 2018, Sun et al., 2018b, Chua et al., 2018, Sanchez-Gonzalez et al., 2018, Pascanu et al., 2017, Khansari-Zadeh and Billard, 2011, Luo et al., 2018] and the reference therein for recent work on model-based RL. Off-policy reinforcement learning…”
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap (2021, background): “…of problems, from getting cars to drive themselves (Pomerleau 1989), to achieving superhuman performance in games like Go (Silver et al. 2016, Sun et al. 2018), to sample-efﬁcient learning of control policies for high DoF robots (Levine and Koltun 2013), to allowing human operators to…”
Predictor-Corrector Policy Optimization (2018, methodology): “4 Next we consider the pure model-based approach with a model that is potentially learned online (Jacobson & Mayne, 1970; Todorov & Li, 2005; Deisenroth & Rasmussen, 2011; Pan & Theodorou, 2014; Levine & Koltun, 2013; Sun et al., 2018).”
Deep Reinforcement Learning (2018, background): “s by watching YouTube (Chapter10). This may give us more motivation and encouragements. As a related work,Branavan et al.(2012) propose to learn strategy games by reading manuals. With achievements inSun et al. (2018) andPang et al.(2018), it is interesting to watch if hierarchical RL approaches in ”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “GPS (Levine & Abbeel, 2014; Mordatch & Todorov, 2014), DPI (Sun et al., 2018b), and related approaches (Kumar et al., 2016) learn a time varying linear model and perform a KL-constrained policy improvement step.”