Taming the Noise in Reinforcement Learning via Soft Updates

By Roy Fox, Ari Pakman, Naftali Tishby. 2015

377 citations recorded by the source corpus.

Model-free reinforcement learning algorithms, such as Q-learning, perform poorly in the early stages of learning in noisy environments, because much effort is spent unlearning biased estimates of the state-action value function. The bias results from selecting, among several noisy estimates, the apparent optimum, which may actually be suboptimal. We propose G-learning, a new off-policy learning algorithm that regularizes the value estimates by penalizing deterministic policies in the beginning of the learning process. We show that this method reduces the bias of the value-function estimation, leading to faster convergence to the optimal value and the optimal policy. Moreover, G-learning enables the natural incorporation of prior domain knowledge, when available. The stochastic nature of G-learning also makes it avoid some exploration costs, a property usually attributed only to on-policy algorithms. We illustrate these ideas in several examples, where G-learning results in significant improvements of the convergence rate and the cost of the learning process.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, methodology): “This formulation corresponds to G-learning (Fox et al., 2015), which can thus be seen as a special case of generalized Ψ-learning.”
GAIT: A Geometric Approach to Information Theory (2019, background): “…ubiquitous in modern machine learning: from variable selection via information gain in decision trees (Ben-David and Shalev-Shwartz, 2014), to using entropy as a regularizer in reinforcement learning (Fox et al., 2016), to rate-distortion theory for training generative models (Alemi et al., 2018).”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “…objective for the inﬁnite horizon discounted case is more involved (Thomas, 2014) and is deferred to Appendix A. Prior methods have proposed directly solving for the optimal Q-function, from which the optimal policy can be recovered (Ziebart et al., 2008; Fox et al., 2016; Haarnoja et al., 2017).”
Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Other approaches have focused directly on reducing the variance (Anschel et al., 2017), minimizing over-ﬁtting to early high variance estimates (Fox et al., 2016), or through corrective terms (Lee et al., 2013).”
Equivalence Between Policy Gradients and Soft Q-Learning (2017, methodology): “We shall consider an entropy-regularized version of the reinforcement learning problem, following various prior work (Ziebart [2010], Fox et al. [2015], Haarnoja et al. [2017], Nachum et al.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “More recently Fox et al. (2016) use a log-weighted-average-exp and Asadi & Littman (2016) propose a mellowmax operator, defined as log-averageexp.”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “See Appendix A.2 as well as (Fox et al., 2016).”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “An alternative is to use KL-divergence instead of entropy as a regularizer, or as a constraint on how much deviation is permitted from a prior policy (Bagnell & Schneider, 2003; Peters et al., 2010; Schulman et al., 2015; Fox et al., 2015).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “, 2017a), ensuring tractability (Todorov, 2006), and mitigating observation noise (Rubin et al., 2012; Fox et al., 2016).”
Boosting the Actor with Dual Critic (2017, methodology): “Their algorithm is based on the duality of entropy-regularized Bellman equation (Todorov, 2007; Rubin et al., 2012; Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), rather than the exact Bellman optimality equation used in our work.”
Smoothed Dual Embedding Control (2017, background): “, 2017), taming the noises in observations (Rubin et al., 2012; Fox et al., 2015), and tractability (Todorov, 2007).”
Towards Characterizing Divergence in Deep Q-Learning (2019, background): “These include target networks (Mnih et al., 2015), entropy regularization (Fox et al., 2016; Haarnoja et al., 2018b), n-step learning (Hessel et al., 2017), and approximate double-Q learning (van Hasselt et al., 2016).”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, neutral): “The update rule we derive is similar to that used in ‘soft’ Q-learning (so-called since the ‘hard’ max is replaced with a soft-max) [6, 16, 18, 30, 47].”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “While there are some works which use this representation of a policy more explicitly (Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), a main disadvantage is that the normalization term may not be tractable for continuous actions, and so computing the policy distribution π ebm…”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “…on how to design effective RL exploration policies (Weng, 2020), in this work we used a simple UCB-style exploration criterion (Chen et al., 2017) as follows: explore ( s, a ) = mean Given N sampled actions from the modiﬁed proposal distribution, we take the action with highest Q explore.”
Maximum a Posteriori Policy Optimisation (2018, methodology): “ harder to optimise lower bound. As an alternative to these policy gradient inspired algorithms, the class of recent algorithms for soft Q-learning (e.g. Rawlik et al. (2012); Haarnoja et al. (2017); Fox et al. (2016) parameterise and estimate a so called “soft” Q-function directly, implicitly induc”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “, 2018c), we introduced the Soft Actor-Critic (SAC) algorithm based on the maximum entropy framework (Ziebart et al., 2008; Toussaint, 2009; Rawlik et al., 2012; Fox et al., 2016; Haarnoja et al., 2017).”
Distral: Robust multitask reinforcement learning (2017, methodology): “The distilled policy π 0 can be interpreted as a policy prior, a perspective well-known in the literature on RL as probabilistic inference [28, 13, 21, 8].”
Multiagent Soft Q-Learning (2018, background): “One of the problems is that at the early stage of learning, the max operator can bring bias into the Q-value estimation (Fox, Pakman, and Tishby 2016).”