OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning

By Peter Henderson, Wei-Di Chang, Pierre-Luc Bacon, David Meger, Joelle Pineau, Doina Precup. 2017

77 citations recorded by the source corpus.

Reinforcement learning has shown promise in learning policies that can solve complex problems. However, manually specifying a good reward function can be difficult, especially for intricate tasks. Inverse reinforcement learning offers a useful paradigm to learn the underlying reward function directly from expert demonstrations. Yet in reality, the corpus of demonstrations may contain trajectories arising from a diverse set of underlying reward functions rather than a single one. Thus, in inverse reinforcement learning, it is useful to consider such a decomposition. The options framework in reinforcement learning is specifically designed to decompose policies in a similar light. We therefore extend the options framework and propose a method to simultaneously recover reward options in addition to policy options. We leverage adversarial methods to learn joint reward-policy options using only observed expert states. We show that this approach works well in both simple and complex continuous control tasks and shows significant performance increases in one-shot transfer learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “The last baseline is representative of a class of HIL algorithms [23, 24, 26], which are difﬁcult to ﬁne-tune because it is not clear how to provide rewards for improving low-level primitives.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, background): “Henderson et al. (2018) and Torabi et al. (2018b) extend Generative Adversarial Imitation Learning (Ho & Ermon, 2016) to remove the need for action labels.”
Sphere Generative Adversarial Network Based on Geometric Moment Matching (2019, methodology): “[8], generative adversarial networks (GANs) have attracted much research interest, and they have been used to achieve outstanding performance in a wide range of computer vision applications including in image generation [17, 26], super resolution [14], video prediction [19], style transfer [5, 12, 3”
Recent Advances in Imitation Learning from Observation (2019, methodology): “Another algo-rithm, called OptionGAN [Henderson et al. , 2018], uses the same algorithm combined with option learning to enable the agent to decompose policies for cases in which the demonstrations contain trajectories result from a diverse set of underlying reward functions rather than a single one”
Improving GAN Training via Binarized Representation Entropy (BRE) Regularization (2018, background): “…high dimensional data in the last two years, with successful applications on image data (Isola et al., 2016; Shrivastava et al., 2016), and high potential for predictive representation learning (Mathieu et al., 2015) as well as reinforcement learning (Finn et al., 2016; Henderson et al., 2017).”
Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018, methodology): “Future extensions of our work may involve learning a distributional reward estimator as in [39], off-policy experience replay for the reward estimator, or learn options for reward estimators as in [40].”