Learning Robust Options

By Daniel Jaymin Mankowitz, Timothy A. Mann, Pierre-Luc Bacon, Doina Precup, Shie Mannor. 2018

51 citations recorded by the source corpus.

Robust reinforcement learning aims to produce policies that have strong guarantees even in the face of environments/transition models whose parameters have strong uncertainty. Existing work uses value-based methods and the usual primitive action setting. In this paper, we propose robust methods for learning temporally abstract actions, in the framework of options. We present a Robust Options Policy Iteration (ROPI) algorithm with convergence guarantees, which learns options that are robust to model uncertainty. We utilize ROPI to learn robust options with the Robust Options Deep Q Network (RO-DQN) that solves multiple tasks and mitigates model misspecification due to model uncertainty. We present experimental results which suggest that policy iteration with linear features may have an inherent form of robustness when using coarse feature representations. In addition, we present experimental results which demonstrate that robustness helps policy iteration implemented on top of deep neural networks to generalize over a much broader range of dynamics than non-robust policy iteration.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

Learning to Learn without Forgetting By Maximizing Transfer and Minimizing Interference (2018, background): “The options framework has also been considered as a solution to a similar continual RL setting to the one we explore (Mankowitz et al., 2018).”
Challenges of Real-World Reinforcement Learning (2019, background): “This has previously been applied in a number of different works (Mankowitz et al., 2018; Di Castro et al., 2012; Derman et al., 2018).”
Unicorn: Continual Learning with a Universal, Off-policy Agent (2018, background): “Additions include optimizing for risk-aware, [46], robust [45, 27] as well as exploratory [6, 31] objectives to yield improved performance.”
DREAM Architecture: a Developmental Approach to Open-Ended Learning in Robotics (2020, background): “It can accelerate learning [107] or make learning more robust [108].”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “Recently, a number of works have surfaced that have shown this formulation to yield robust policies that are agnostic to a range of perturbations in the environment [Tamar et al. 2014; Mankowitz et al. 2018a; Shashua and Mannor 2017; Derman et al. 2018; 2019; Mankowitz et al. 2019].”
Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations (2020, background): “RMDP theory has inspired robust deep Q-learning [62] and policy gradient algorithms [41, 12, 42] that are robust against small environmental changes.”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Previous research in this area includes learning the options from scratch (Mankowitz et al. 2016a, b; Bacon et al. 2017) as well as planning, given a pre-trained set of options (Schaul et al. 2015; Mankowitz et al. 2018b).”