Bridging the Gap Between Value and Policy Based Reinforcement Learning

By Ofir Nachum, Mohammad Norouzi, Kelvin Xu, Dale Schuurmans. 2017

535 citations recorded by the source corpus.

We establish a new connection between value and policy based reinforcement learning (RL) based on a relationship between softmax temporal value consistency and policy optimality under entropy regularization. Specifically, we show that softmax consistent action values correspond to optimal entropy regularized policy probabilities along any action sequence, regardless of provenance. From this observation, we develop a new RL algorithm, Path Consistency Learning (PCL), that minimizes a notion of soft consistency error along multi-step action sequences extracted from both on- and off-policy traces. We examine the behavior of PCL in different scenarios and show that PCL can be interpreted as generalizing both actor-critic and Q-learning algorithms. We subsequently deepen the relationship by showing how a single model can be used to represent both a policy and the corresponding softmax state values, eliminating the need for a separate critic. The experimental evaluation demonstrates that PCL significantly outperforms strong actor-critic and Q-learning baselines across several benchmarks.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “In the case of path consistency methods (Nachum et al., 2017a,b), the maximum entropy framework offers an appealing mechanism for off-policy learning.”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017).”
Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, methodology): “The existence of multi-step softmax consistencies has been noted by prior work - ﬁrst by Nachum et al. (2017) in the presence of entropy regularization.”
Equivalence Between Policy Gradients and Soft Q-Learning (2017, methodology): “• O’Donoghue et al. [2016] begin with a similar motivation as the current paper: that a possible explanation for Q-learning and SARSA is that their updates are similar to policy gradient updates. They decompose the Q-function into a policy part and a value part, inspired by dueling Q-networks (Wang ”
Self-Imitation Learning (2018, background): “…recent studies on the relationship between policy gradient and Q-learning have shown that entropy-regularized policy gradient and Q-learning are closely related or even equivalent depending on assumptions (Nachum et al., 2017; O’Donoghue et al., 2017; Schulman et al., 2017a; Haarnoja et al., 2017).”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “We make the following additional practical modiﬁcations: • As entropy regularization has been shown to be important on these tasks (Nachum et al., 2017b; Haarnoja et al., 2018), we augment the rewards with a causal entropy term; i.e. , replace r in (25) with r − τ log π ( a 0 | s 0 ), where τ is…”
Generalized Off-Policy Actor-Critic (2019, background): “The target policy can then be derived from the value function directly (O’Donoghue et al., 2016; Nachum et al., 2017a; Schulman et al., 2017a).”
Boosting the Actor with Dual Critic (2017, methodology): “Their algorithm is based on the duality of entropy-regularized Bellman equation (Todorov, 2007; Rubin et al., 2012; Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), rather than the exact Bellman optimality equation used in our work.”
Smoothed Dual Embedding Control (2017, methodology): “In contrast to our saddle point optimization approach (10), Nachum et al. (2017a) considered a different way to handle the double sampling issue by solving instead an upper bound of (9), namely, minV,π∈P f̃(V, π) := Es,a,s′ [ (R(s, a) + γV (s′)− λ log π(a|s)− V (s))(2) ] .”
Variational Bayesian Reinforcement Learning with Regret Bounds (2018, neutral): “The update rule we derive is similar to that used in ‘soft’ Q-learning (so-called since the ‘hard’ max is replaced with a soft-max) [6, 16, 18, 30, 47].”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, methodology): “While there are some works which use this representation of a policy more explicitly (Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), a main disadvantage is that the normalization term may not be tractable for continuous actions, and so computing the policy distribution π ebm…”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “…on how to design effective RL exploration policies (Weng, 2020), in this work we used a simple UCB-style exploration criterion (Chen et al., 2017) as follows: explore ( s, a ) = mean Given N sampled actions from the modiﬁed proposal distribution, we take the action with highest Q explore.”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, background): “Another approach is entropy regularization (Ziebart et al., 2008; Haarnoja et al., 2018; Nachum et al., 2017; Williams & Peng, 1991), where causal entropy is added to the rewards, making the Q-function smoother and facilitating optimization (Ahmed et al.”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “Efforts to combine the ease-of-use of on-policy algorithms with the sample efficiency of off-policy algorithms have been fruitful (Gu et al., 2016; O’Donoghue et al., 2016b; Wang et al., 2016; Gu et al., 2017; Nachum et al., 2017; Degris et al., 2012).”
Reinforcement Learning from Imperfect Demonstrations (2018, methodology): “• Path Consistency Learning (PCL): the PCL (Nachum et al., 2017) method that minimizes the soft path consistency loss.”
Diversity is All You Need: Learning Skills without a Reward Function (2018, background): “Related work has also examined connections between RL and information theory (Ziebart et al., 2008; Schulman et al., 2017; Nachum et al., 2017; Haarnoja et al., 2017) and developed maximum entropy algorithms with these ideas Haarnoja et al. (2018; 2017).”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “Prior work has proposed model-free deep RL algorithms that perform on-policy learning with entropy maximization (O’Donoghue et al., 2016), as well as off-policy methods based on soft Q-learning and its variants (Schulman et al., 2017a; Nachum et al., 2017a; Haarnoja et al., 2017).”
Distral: Robust multitask reinforcement learning (2017, background): “Instead (1) reduces to an entropy regularized expected returns with entropy regularization factor β (cid:48) = β/ (1 − α ) = 1 /c Ent , so that the optimal policy is of the Boltzmann form with inverse temperature β (cid:48) [21, 8, 24, 18].”
A Brief Survey of Deep Reinforcement Learning (2017, positive): “Meanwhile, the A3C-based agent of Mirowski et al. [82] was additionally trained to construct a depth map given RGB inputs, which helps it in its task of learning to navigate a 3D environment.”