BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning

By Zijian Zhou, Z. Wang, Che Wang, Yanqiu Wu, Qing Deng, Keith W. Ross. 2019

143 citations recorded by the source corpus.

There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL learns a V function, uses the V function to select actions it believes to be high-performing, and then uses those actions to train a policy network using imitation learning. For the MuJoCo benchmark, we provide a comprehensive experimental study of BAIL, comparing its performance to four other batch Q-learning and imitation-learning schemes for a large variety of batch datasets. Our experiments show that BAIL's performance is much higher than the other schemes, and is also computationally much faster than the batch Q-learning schemes.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Offline RL Without Off-Policy Evaluation (2021, positive): “Another idea, proposed by [Wang et al., 2018, Siegel et al., 2020, Wang et al., 2020b, Chen et al., 2020] is to modify an imitation learning algorithm either by ﬁltering or weighting the observed actions to incentivize policy improvement.”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “Other approaches use a weighted version of BC to favor high advantage actions [Wang et al., 2018, Peng et al., 2019, Siegel et al., 2020, Wang et al., 1https://github.com/sfujim/TD3_BC 2020, Nair et al., 2020] or perform BC over an explicit subset of the data [Chen et al., 2020].”
Critic Regularized Regression (2020, methodology): “Last but not least, methods that rely on reward filtering would also favor arm one over arm two [7, 18].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Recommender systems are amongst the most solicited large-scale software systems, and RL proposes an enticing framework for optimizing them (Covington et al. 2016; Chen et al. 2019a).”