OFFER: Off-Environment Reinforcement Learning

By Kamil Andrzej Ciosek, Shimon Whiteson. 2017

54 citations recorded by the source corpus.

Policy gradient methods have been widely applied in reinforcement learning. For reasons of safety and cost, learning is often conducted using a simulator. However, learning in simulation does not traditionally utilise the opportunity to improve learning by adjusting certain environment variables - state features that are randomly determined by the environment in a physical setting but controllable in a simulator. Exploiting environment variables is crucial in domains containing significant rare events (SREs), e.g., unusual wind conditions that can crash a helicopter, which are rarely observed under random sampling but have a considerable impact on expected return. We propose off environment reinforcement learning (OFFER), which addresses such cases by simultaneously optimising the policy and a proposal distribution over environment variables. We prove that OFFER converges to a locally optimal policy and show experimentally that it learns better and faster than a policy gradient baseline.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Stabilising Experience Replay for Deep Multi-Agent Reinforcement Learning (2017, background): “Just as an RL agent can use importance sampling to learn off-policy from data gathered when its own policy was different, so too can it learn off-environment (Ciosek & Whiteson, 2017) from data gathered in a different environment.”
Trial without Error: Towards Safe Reinforcement Learning via Human Intervention (2017, background): “2 This problem could potentially be side-stepped by training in simulation [10].”
A survey and critique of multiagent deep reinforcement learning (2018, neutral): “sampling [216, 217] and off-environment RL [218].”
Expected Policy Gradients (2017, methodology): “2 Robotic Arm We also tested EPG on a variant of a simulated robotic-arm [13] control task to which policy gradient methods have recently been applied [4].”