Smoothed Dual Embedding Control

By Bo Dai, Albert Eaton Shaw, Lihong Li, Lin Xiao, Niao He, Jianshu Chen, Le Song. 2017

25 citations recorded by the source corpus.

We revisit the Bellman optimality equation with Nesterov's smoothing technique and provide a unique saddle-point optimization perspective of the policy optimization problem in reinforcement learning based on Fenchel duality. A new reinforcement learning algorithm, called Smoothed Dual Embedding Control or SDEC, is derived to solve the saddle-point reformulation with arbitrary learnable function approximator. The algorithm bypasses the policy evaluation step in the policy optimization from a principled scheme and is extensible to integrate with multi-step bootstrapping and eligibility traces. We provide a PAC-learning bound on the number of samples needed from one single off-policy sample path, and also characterize the convergence of the algorithm. Finally, we show the algorithm compares favorably to the state-of-the-art baselines on several benchmark control problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Lee and He (2018) extend a line of work applying to convex optimization techniques to policy optimization (Chen and Wang, 2016; Wang and Chen, 2016; Dai et al., 2017a,b) to the off-policy setting.”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “Similarly, for the training of value function, we use primal-dual based optimization methods (Dai et al., 2017b; Feng et al., 2019) to minimize the bellman residual: min φ max fβ∈F 1 |M| ∑ i∈M (( Vφ(si)− π(ai|si) π0(ai|si) (ri + γVφ(s ′ i)) ) fβ(si)− 1 2 fβ(si) 2 ) , where Vφ(s) is the…”
Learning Deep Hidden Nonlinear Dynamics from Aggregate Data (2018, methodology): “Inspired by (Dai et al., 2017), we can see that once we obtain D∗ t , Algorithm 1 can be understood as a special case of stochastic gradient descent for a non-convex problem.”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, methodology): “The convergence analysis of SBEED (Theorem 5 in Dai et al. (2017)) is conducted within a bi-level optimization framework, assuming the exact solution of the inner optimization problem can be obtained.”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, background): “, 2013) or requiring a primal-dual formulation as has been done for control (Dai et al., 2017).”