UCB EXPLORATION VIA Q-ENSEMBLES

By Richard Y. Chen, Szymon Sidor, Pieter Abbeel, John Schulman. 2018

151 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Continuous Doubly Constrained Batch Reinforcement Learning (2021, background): “To correct extra-overestimation within each of the M individual Q-networks, Algorithm 1 actually applies a separate extra-overestimation penalty ∆j specific to each Q-network.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “While there exists a rich literature on how to design effective RL exploration policies (Weng, 2020), in this work we used a simple UCB-style exploration criterion (Chen et al., 2017) as follows:”
Evolved Policy Gradients (2018, background): “Prior works add auxiliary reward terms that emphasize exploration (4; 12; 21; 40; 2; 22) and entropy loss terms (20; 29; 9; 15).”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “Intrinsic rewards can be derived from various measures: novelty [43, 55, 6, 56], prediction-error [38, 3], ensemble disagreement [11, 39, 48, 46, 18, 50], or information gain [23].”
Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble (2021, methodology): “Following this work, the UCB exploration algorithm [5] constructs an upper confidence bound [3] of the Q-values using the empirical mean and standard deviation of Q-ensembles, which is used to promote efficient exploration by applying the principle of optimism in the face of uncertainty [7].”
Learning Montezuma's Revenge from a Single Demonstration (2018, methodology): “Prior works (Dearden et al. [1999], Kolter and Ng [2009], Tang et al.”
Towards Better Interpretability in Deep Q-Networks (2018, methodology): “The Q-ensemble agent using UCBstyle exploration is trained for up to 40M frames (Chen et al. 2018).”
QUOTA: The Quantile Option Architecture for Reinforcement Learning (2018, methodology): “Different approaches are used to achieve this optimism, e.g., count-based methods (Auer 2002; Kocsis and Szepesv ´ ari 2006; Bellemare et al. 2016; Ostrovski et al. 2017; Tang et al. 2017) and Bayesian methods (Kaufmann, Capp ´ e, and Garivier 2012; Chen et al. 2017; O’Donoghue et al. 2017).”
Distributional Reinforcement Learning for Efficient Exploration (2019, methodology): “Both count-based methods (Auer, 2002; Kaufmann et al., 2012; Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017) and Bayesian methods (Kaufmann et al., 2012; Chen et al., 2017; O’Donoghue et al., 2017) follow this optimism principle.”
Provable Rich Observation Reinforcement Learning with Combinatorial Latent States (2021, background): “Empirical results (Tang et al., 2017; Chen et al., 2017; Bellemare et al., 2016; Pathak et al., 2017) have achieved inspiring performance on several RL benchmarks, while theoretical works (Krishnamurthy et al.”