Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization

By Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, Shixiang Shane Gu. 2020

166 citations recorded by the source corpus.

Most reinforcement learning (RL) algorithms assume online access to the environment, in which one may readily interleave updates to the policy with experience collection using that policy. However, in many real-world applications such as health, education, dialogue agents, and robotics, the cost or potential risk of deploying a new data-collection policy is high, to the point that it can become prohibitive to update the data-collection policy more than a few times during learning. With this view, we propose a novel concept of deployment efficiency, measuring the number of distinct data-collection policies that are used during policy learning. We observe that naively applying existing model-free offline RL algorithms recursively does not lead to a practical deployment-efficient and sample-efficient algorithm. We propose a novel model-based algorithm, Behavior-Regularized Model-ENsemble (BREMEN) that can effectively optimize a policy offline using 10-20 times fewer data than prior works. Furthermore, the recursive application of BREMEN is able to achieve impressive deployment efficiency while maintaining the same or better sample efficiency, learning successful policies from scratch on simulated robotic environments with only 5-10 deployments, compared to typical values of hundreds to millions in standard RL baselines. Codes and pre-trained models are available at this https URL .

Subfields: Artificial Intelligence, Computer Science Applications, Mechanical Engineering, Reinforcement Learning, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Mechanical Engineering

In Reinforcement Learning

In ML Systems & Hardware

Citations in the corpus

A Minimalist Approach to Offline Reinforcement Learning (2021, background): “…of offline RL for safety-critical real-world use cases [Mandel et al., 2014, Gottesman et al., 2018, Gauci et al., 2018, Jaques et al., 2019, Matsushima et al., 2020] as well as the current trend of reporting only the mean-value of the final policy in offline benchmarking [Fu et al., 2021].”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “These regularizers have appeared as divergence penalties (Kumar et al., 2019; Jaques et al., 2019; Wu et al., 2019), implicitly through appropriate network initializations (Matsushima et al., 2020), or more explicitly through careful parameterization of the policy (Fujimoto et al., 2019).”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “We refer to online RL as the setting where iteratively, a batch of M environment steps with an exploration policy are inter-leaved with M RL updates (Levine et al., 2020; Matsushima et al., 2020).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Model-based offline RL methods (Finn & Levine, 2017; Ebert et al., 2018; Kahn et al., 2018; Kidambi et al., 2020; Yu et al., 2020c; Matsushima et al., 2020; Argenson & DulacArnold, 2020; Swazinna et al., 2020; Rafailov et al., 2020; Lee et al., 2021) provide an alternative approach to policy learnin”
Representation Matters: Offline Pretraining for Sequential Decision Making (2021, background): “Accordingly, much of the recent oﬄine RL research focuses on proposing new policy optimization algorithms amenable to learning from oﬄine datasets (e.g., Fujimoto et al. (2019); Wu et al. (2019); Agarwal et al. (2020); Kumar et al. (2020); Yu et al. (2020); Matsushima et al. (2020)).”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, background): “For example, a robot learning through trial and error in the real world requires costly human supervision, safety checks, and resets (Atkeson et al., 2015), rendering many standard online RL algorithms inapplicable (Matsushima et al., 2020).”
Representation Balancing Offline Model-based Reinforcement Learning (2021, methodology): “Matsushima et al. (2020), MOOSE (Swazinna et al., 2020) and MBOP (Argenson & Dulac-Arnold, 2020) guide their policy optimization using the behavior policy, similar to the behavior regularization approaches.”
Offline Reinforcement Learning from Images with Latent Space Models (2020, background): “…al., 2019), approximate dynamic programming algorithms (Fujimoto et al., 2018; Kumar et al., 2019, 2020; Liu et al., 2020; Agarwal et al., 2019), and model-based RL algorithms (Kidambi et al., 2020; Yu et al., 2020b; Argenson and Dulac-Arnold, 2020; Matsushima et al., 2020; Swazinna et al., 2020).”