SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation

By Bo Dai, Albert Eaton Shaw, Lihong Li, Lin Xiao, Niao He, Z. Liu, Jianshu Chen, Le Song. 2017

305 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Lee and He (2018) extend a line of work applying to convex optimization techniques to policy optimization (Chen and Wang, 2016; Wang and Chen, 2016; Dai et al., 2017a,b) to the off-policy setting.”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, methodology): “However, as discussed in the literature (Antos et al., 2008; Lazaric et al., 2012; Dai et al., 2018; Nachum et al., 2019), using the blocking technique of Yu (1994), the statistical error provided here can be generalized to β-mixing samples in a single sample path.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “Other works (Chen and Wang, 2016; Wang, 2017; Dai et al., 2018b; Chen et al., 2018) consider the (augmented) Lagrangian of the LP characterization of Bellman optimality for the optimal V -function, which is slightly diﬀerent from the LP characterization with respect to the optimal Q -function we…”
Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “Similarly, for the training of value function, we use primal-dual based optimization methods (Dai et al., 2017b; Feng et al., 2019) to minimize the bellman residual: min φ max fβ∈F 1 |M| ∑ i∈M (( Vφ(si)− π(ai|si) π0(ai|si) (ri + γVφ(s ′ i)) ) fβ(si)− 1 2 fβ(si) 2 ) , where Vφ(s) is the…”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, methodology): “However, by exploiting blocking technique in Proposition 15 of [53], following [2, 23, 9], all the sample complexity we provided can be easily generalized for single β -mixing sample path, i.e. , { s i , a i , r i , s i } =1 is strictly stationary and mixing in an exponential rate with parameter b,…”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “In the con-trol scenario, convergent algorithms such as SBEED (Dai et al., 2018) and Greedy-GQ (Maei et al., 2010) have been developed.”
Information-Theoretic Considerations in Batch Reinforcement Learning (2019, methodology): “In fact, Dai et al. (2018) derives a closely related algorithm using Fenchel dual and shows that the algorithm is always convergent.”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “It also draws a connection to the recent primal-dual reformulations of the Bellman equation (Dai et al., 2017, 2018b), which formulate V π as a saddle-point of the following minimax problem: min This is equivalent to minimizing L 2 ( V ) as (6), except that the L2 constraint is replaced by a…”
On Oracle-Efficient PAC RL with Rich Observations (2018, positive): “This (G,Π) representation is useful in resolving the computational difficulty in the deterministic setting, and has also been used in practice [32].”
Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning (2021, background): “We also note that the convergence of these methods has been shown for simple scenarios but is still a challenge for more complex scenarios or when function approximators are used (30).”
Kernel Exponential Family Estimation via Doubly Dual Embedding (2018, background): “From such a view, we can prove the sublinearly convergence rate to a stationary point when stepsize is diminishing following Ghadimi and Lan (2013); Dai et al.”
On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach (2019, background): “, 2018) and primal-dual reinforcement learning (Du et al., 2017; Dai et al., 2018), explicitly specify the order of moves between players and the order of which player acts first is crucial for the problem.”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “While much of the modern literature surrounding Bellman error minimization emphasizes the double sampling problem (Dai et al., 2018; Feng et al., 2019; Saleh & Jiang, 2019; Bas-Serrano et al., 2021), our analysis shows much more fundamental problems; Bellman error is not guaranteed to closely…”
Harnessing Inﬁnite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, methodology): “Primal-Dual Value Learning Primal-dual optimization techniques have been widely used for off-policy value function learning and policy optimization (Liu et al., 2015; Chen & Wang, 2016; Dai et al., 2017a,b; Feng et al., 2019).”
Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problems (2020, methodology): “In reinforcement learning, solving Bellman equations can also be reformulated as minimax optimization problems [Chen and Wang, 2016, Dai et al., 2017, 2018].”
Recent advances in reinforcement learning in finance (2021, methodology): “In addition to linear functional approximations, we refer the reader to [58] for a general discussion of RL with nonlinear functional approximations and to Section 3 for neural network approximation, one of the most popular nonlinear functional approximations used in practice.”
A Theory of Regularized Markov Decision Processes (2019, background): “Other approaches are based on a notion of temporal consistency equation, somehow extending the notion of Bellman residual to the regularized case (Nachum et al., 2017; Dai et al., 2018; Nachum et al., 2018), or on policy gradient (Williams, 1992; Mnih et al.”