EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL

By Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, Dale Schuurmans, Shixiang Shane Gu. 2020

146 citations recorded by the source corpus.

Off-policy reinforcement learning (RL) holds the promise of sample-efficient learning of decision-making policies by leveraging past experience. However, in the offline RL setting -- where a fixed collection of interactions are provided and no further interactions are allowed -- it has been shown that standard off-policy RL methods can significantly underperform. Recently proposed methods aim to address this shortcoming by regularizing learned policies to remain close to the given dataset of interactions. However, these methods involve several configurable components such as learning a separate policy network on top of a behavior cloning actor, and explicitly constraining action spaces through clipping or reward penalties. Striving for simultaneous simplicity and performance, in this work we present a novel backup operator, Expected-Max Q-Learning (EMaQ), which naturally restricts learned policies to remain within the support of the offline dataset \emph{without any explicit regularization}, while retaining desirable theoretical properties such as contraction. We demonstrate that EMaQ is competitive with Soft Actor Critic (SAC) in online RL, and surpasses SAC in the deployment-efficient setting. In the offline RL setting -- the main focus of this work -- through EMaQ we are able to make important observations regarding key components of offline RL, and the nature of standard benchmark tasks. Lastly but importantly, we observe that EMaQ achieves state-of-the-art performance with fewer moving parts such as one less function approximation, making it a strong, yet easy to implement baseline for future work.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “Others have tackled this problem using deep learning, but have not used KL-control (Liu et al., 2019; Gelada and Bellemare, 2019; Bhatt et al., 2019; Kumar et al., 2019; Agarwal et al., 2019; Fujimoto et al., 2018; Ghasemipour et al., 2020).”
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation (2021, neutral): “Using the perturbation actor to modify the samples is optional – we find that removing it is often beneficial (see Appendix I), as in other prior work [42].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “This may be implemented using an estimate of the behavior policy and then defining an explicit policy parameterization [Fujimoto et al., 2019b, Ghasemipour et al., 2020] or by using divergence regularization [Jaques et al., 2019, Kumar et al., 2019, Wu et al., 2019, Siegel et al., 2020, Guo et al.,…”
Regularized Behavior Value Estimation (2021, background): “More recent offline RL approaches fall into three broad categories: 1) policyconstraint approaches regularize the learned policy to stay close to the behavior policy either explicitly or implicitly (Fujimoto et al., 2019b; Kumar et al., 2019; Jaques et al., 2019; Siegel et al., 2020; Wang et al., 20”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “One of the limitations of this approach is that a large number of sampled actions might be required for competitive performance (Ghasemipour et al., 2021).”
Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “In blue, we highlight the only modifications CDC makes to a standard off-policy actor-critic framework that has been suitably adapted for continuous batch RL via the aforementioned techniques like EMaQ [18] and lower-confidence bounds for Q-”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “…either implicitly via regularized variants of importance sampling based algorithms [47, 58, 35, 59, 41], ofﬂine actor-critic methods [53, 45, 27, 16, 64], applying uncertainty quantiﬁcation to the predictions of the Q-values [2, 28, 63, 34], and learning conservative Q-values [29, 55] or…”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “Results: As shown in Table 1, CQL+OPAL outperforms nearly all the baselines on antmaze (see Figure 1 and Figure 3 for visualization) and kitchen tasks, with the exception of EMAQ having similar performance on kitchen mixed.”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “Numerous algorithmic work ensued (Wu et al., 2019; Jaques et al., 2020; Ghasemipour et al., 2021; Kumar et al., 2020; Fujimoto & Gu, 2021) with various applications (Jaques et al., 2020; Chebotar et al., 2021).”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, methodology): “Examples include model-based methods that learn pessimistic MDPs [37], [39], [40], model-free methods that reduce the Q-functions on unseen state-action pairs [41]–[43], and policy-based methods that minimize the divergence between the learned policy and the behavior policy [20], [21], [44]–[49].”
Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “…over actions that serve to constrain the learned policy to avoid out-of-distribution behavior that is not supported under the dataset [16, 31, 17]; imitation learning, where it is used to ﬁt an expert’s actions to obtain a policy [50, 51]; and other areas such as hierarchical RL [46, 11, 26].”
IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, unknown): “To this end, recent methods use implicit divergence constraints [38, 37, 33, 45], explicit density models [47, 13, 27, 15], and supervised learning terms [12].”
Offline RL for Natural Language Generation with Implicit Language Q Learning (2022, methodology): “We train models with CQL α = {0.1, 1.0, 10.0}, and we evaluate each model with the number of EMAQ-style [29] samples chosen from N={4, 8, 16}.”