Learning from Conditional Distributions via Dual Embeddings

By Bo Dai, Niao He, Yunpeng Pan, Byron Boots, Le Song. 2016

136 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “(10) They further apply tricks inspired from dual embeddings (Dai et al., 2016) to make the objective tractable, and to avoid the bias caused due to sampled estimates.”
Black-box Off-policy Estimation for Infinite-Horizon Reinforcement Learning (2020, background): “2 BLACK-BOX ESTIMATOR WITH MMD There are different choices for D(· ‖ ·) in equation 9, and multiple approaches to solve it (e.g., Nguyen et al., 2010; Dai et al., 2017).”
Batch Stationary Distribution Estimation (2020, background): “Crucially, the dual variable v is a scalar, making this problem much simpler than dual embedding (Dai et al., 2017), where the dual variables form a parameterized function that introduces approximation error.”
GenDICE: Generalized Offline Estimation of Stationary Values (2020, methodology): “(12) Applying the interchangeability principle (Shapiro et al., 2014; Dai et al., 2016), one can replace the inner max in the first term over scalar f to maximize over a function f (·, ·) : S ×A→ R min τ≥0 max f :S×A→R,u∈R J (τ, u, f) = (1− γ)Eμ0π [f (s, a)] + γETp [τ (s, a) f (s′, a′)] − Ep [τ (s, ”
GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “…address a double sampling issue, Zhang et al. (2020) , where φ ∗ is the Fenchel conjugate of φ , and use the interchangeability principle for interchanging maximization and expectation (Shapiro et al., 2014; Dai et al., 2016), yielding the following problem, which is equivalent to (2): where Here”
Reinforcement Learning via Fenchel-Rockafellar Duality (2020, methodology): “Typically, the Q-values are estimated by minimizing squared Bellman residuals (or some variation on this) (Sutton et al., 2008, 2009; Liu et al., 2015; Dai et al., 2016; Du et al., 2017); i.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “Here, we apply the dual embedding technique (Dai et al., 2016, 2018b).”
DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Corrections (2019, neutral): “By the interchangeability principle [6, 34, 36], we may replace the inner max over scalar ζ to a max over functions ζ : S ×A→ R and obtain a min-max saddle-point optimization:”
Smoothed Dual Embedding Control (2017, methodology): “Specifically, by exploiting the Fenchel dual of the square function, i.e., x2 = maxν ( 2νx− ν2 ) , and further applying the interchangeability principle (Dai et al., 2016, Lemma 1), we can show that (9) is equivalent to the saddle point problem min V,π∈P max ν∈F(S×A) L(V, π; ν) := 2Es,a,s′ [ν(s, a)…”
A Kernel Loss for Solving the Bellman Equation (2019, methodology): “It also draws a connection to the recent primal-dual reformulations of the Bellman equation (Dai et al., 2017, 2018b), which formulate V π as a saddle-point of the following minimax problem: min This is equivalent to minimizing L 2 ( V ) as (6), except that the L2 constraint is replaced by a…”
Off-Policy Evaluation via the Regularized Lagrangian (2020, background): “Therefore, the vanilla stochastic gradient descent is no longer applicable [5], due to the bias in the gradient estimator.”
Kernel Exponential Family Estimation via Doubly Dual Embedding (2018, background): “From such a view, we can prove the sublinearly convergence rate to a stationary point when stepsize is diminishing following Ghadimi and Lan (2013); Dai et al. (2017).”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, unknown): “The first approach reformulates the Bellman error as a saddle-point optimization problem where a new concave function ν : S × A → R is introduced such that the loss becomes L(Q̂ , ν) = 2Es,a,s′ [ν(s, a)(Q̂(s, a) − R(s, a) − γ maxa′ Q̂(s ′, a ′))] − Es,a,s′ [ν(s, a)2] [53].”
Offline Policy Selection under Uncertainty (2020, methodology): “This formulation can be equivalently rewritten as Then, by applying Markov’s inequality, i.e. , P q ( (cid:96) ( ζ ) we can obtain an upper bound on (12) as where the equality follows by interchangeability (Shapiro et al., 2014; Dai et al., 2017).”
Stochastic Variance Reduction Methods for Policy Evaluation (2017, methodology): “There are other algorithms whose complexity scale linearly with n and d , including iLSTD (Geramifard et al., 2007), and TDC (Sutton et al., 2009b), fLSTD-SA (Prashanth et al., 2014), and the more recent algorithms of Wang et al. (2016) and Dai et al. (2016).”
Global Convergence and Variance-Reduced Optimization for a Class of Nonconvex-Nonconcave Minimax Problems (2020, unknown): “In reinforcement learning, solving Bellman equations can also be reformulated as minimax optimization problems [Chen and Wang, 2016, Dai et al., 2017, 2018].”
Scalable Bilinear π Learning Using State and Action Features (2018, background): “It should be noted that there is another line of research that studies saddle-point formulations that result from the ﬁxed-point (as opposed to LP) view of the Bellman equation for ﬁxed policy evaluation (Mahadevan et al., 2014; Macua et al., 2015; Du et al., 2017; Dai et al., 2017).”