Boosting the Actor with Dual Critic

By Bo Dai, Albert Eaton Shaw, Niao He, Lihong Li, Le Song. 2017

46 citations recorded by the source corpus.

This paper proposes a new actor-critic-style algorithm called Dual Actor-Critic or Dual-AC. It is derived in a principled way from the Lagrangian dual form of the Bellman optimality equation, which can be viewed as a two-player game between the actor and a critic-like function, which is named as dual critic. Compared to its actor-critic relatives, Dual-AC has the desired property that the actor and dual critic are updated cooperatively to optimize the same objective function, providing a more transparent way for learning the critic that is directly related to the objective function of the actor. We then provide a concrete algorithm that can effectively solve the minimax optimization problem, using techniques of multi-step bootstrapping, path regularization, and stochastic dual ascent algorithm. We demonstrate that the proposed algorithm achieves the state-of-the-art performances across several benchmarks.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Ofir Nachum
3. Bo Dai
4. Lihong Li
5. George Tucker
6. Aviral Kumar
7. Pieter Abbeel
8. Ilya Kostrikov

In Management Science and Operations Research

1. Ofir Nachum
2. Sergey Levine
3. Bo Dai
4. Aviral Kumar
5. Lihong Li
6. George Tucker
7. Ilya Kostrikov
8. Justin Fu

In Reinforcement Learning

1. Sergey Levine
2. Ofir Nachum
3. Aviral Kumar
4. George Tucker
5. Bo Dai
6. Lihong Li
7. Ilya Kostrikov
8. Justin Fu

Citations in the corpus

• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Lee and He (2018) extend a line of work applying to convex optimization techniques to policy optimization (Chen and Wang, 2016; Wang and Chen, 2016; Dai et al., 2017a,b) to the off-policy setting.”
• Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “Note that similar forms of Bπ have appeared in the literature, usually used to encode constraints in a dual linear program for an MDP (e.g., Wang et al., 2007; Wang, 2017; Dai et al., 2018).”
• Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “Depending on the specific regularization employed, deriving the optimal policy from V ∗, K∗ may be difficult in 7The Lagrangian of (72) or (68) may also be used to derive several existing tabular (Chen and Wang, 2016) and on-policy algorithms (Wang, 2017; Dai et al., 2017; Chen et al., 2018).”
• AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “…for many different purposes in the literature, including exploration (de Farias and Van Roy, 2000; Haarnoja et al., 2017, 2018), smoothing (Dai et al., 2018b), avoiding premature convergence (Nachum et al., 2017a), ensuring tractability (Todorov, 2006), and mitigating observation noise…”
• Smoothed Dual Embedding Control (2017, methodology): “ as we discussed in Section 3, the PCL algorithms becomes biased in stochastic environment, which may lead to inferior solutions Baird (1995). Several recent works (Chen & Wang, 2016; Wang, 2017; Dai et al., 2018) have also considered saddle-point formulations of Bellman equations, but these for”
• A Kernel Loss for Solving the Bellman Equation (2019, methodology): “It also draws a connection to the recent primal-dual reformulations of the Bellman equation (Dai et al., 2017, 2018b), which formulate V π as a saddle-point of the following minimax problem: min This is equivalent to minimizing L 2 ( V ) as (6), except that the L2 constraint is replaced by a…”
• Scalable Bilinear π Learning Using State and Action Features (2018, background): “Dai et al. (2018) considers a related yet different primal-dual formulation that also allows compact representations, but no rate of convergence is provided.”
• A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, unknown): “…the reward and cost from the perspective of an optimization criterion, in which they consider the safety constraints via the duality property [162], [163], [164], [165] with regard to the state density functions, rather than the cost functions, reward functions and value functions [3], [52],…”
• Stochastic Primal-Dual Q-Learning Algorithm For Discounted MDPs (2019, methodology): “The primal-dual optimization perspective is further employed in [16] with nonlinear function approximations to solve Markov decision problems with continuous state-actions spaces.”
• Neural Approaches to Conversational AI (2018, unknown): “8. Interested readers can refer to a few related works and the references therein for further details (Kakade, 2001; Peters et al., 2005; Schulman et al., 2015a,b; Mnih et al., 2016; Gu et al., 2017; Dai et al., 2018a; Liu et al., 2018b). 2.3.3 Exploration So far we have described basic algorithms f”
• A Review of Safe Reinforcement Learning: Methods, Theory and Applications (2022, methodology): “[222] propose the DCRL method that is to optimise the reward and cost from the perspective of an optimisation criterion based on optimisation theory, in which they consider the safety constraints via the duality property [65, 196, 197, 268] with regard to the state density functions, rather than the”