Learning from Conditional Distributions via Dual Kernel Embeddings

By Bo Dai, Niao He, Yunpeng Pan, Byron Boots, Le Song. 2016

21 citations recorded by the source corpus.

In many machine learning problems, such as policy evaluation in reinforcement learning and learning with invariance, each data point $x$ itself is a conditional distribution $p(z|x)$, and we want to learn a function $f$ which links these conditional distributions to target values $y$. The learning problem becomes very challenging when we only have limited samples or in the extreme case only one sample from each conditional distribution $p(z|x)$. Commonly used approaches either assume that $z$ is independent of $x$, or require an overwhelmingly large sample size from each conditional distribution. To address these challenges, we propose a novel approach which reformulates the original problem into a min-max optimization problem. In the new view, we only need to deal with the kernel embedding of the joint distribution $p(z,x)$ which is easy to estimate. Furthermore, we design an efficient learning algorithm based on mirror descent stochastic approximation, and establish the sample complexity for learning from conditional distributions. Finally, numerical experiments in both synthetic and real data show that our method can significantly improve over the previous state-of-the-arts.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bo Dai
2. Lihong Li
3. Sergey Levine
4. Qiang Liu
5. Le Song
6. Ziyang Tang
7. Bernhard Schölkopf
8. Niao He

In Management Science and Operations Research

1. Ziyang Tang
2. Bo Dai
3. Lihong Li
4. Qiang Liu
5. Niao He
6. Nathan Kallus
7. Masatoshi Uehara
8. Dengyong Zhou

Citations in the corpus

• Doubly Robust Bias Reduction in Infinite Horizon Off-Policy Estimation (2019, methodology): “Similarly, for the training of value function, we use primal-dual based optimization methods (Dai et al., 2017b; Feng et al., 2019) to minimize the bellman residual: min φ max fβ∈F 1 |M| ∑ i∈M (( Vφ(si)− π(ai|si) π0(ai|si) (ri + γVφ(s ′ i)) ) fβ(si)− 1 2 fβ(si) 2 ) , where Vφ(s) is the…”
• Smoothed Dual Embedding Control (2017, methodology): “oblem in order to bypass the double sampling issue. Specically, by exploiting the Fenchel dual of the square function, i.e., x2 = max 2x 2 ;and further applying the interchangeability principle (Dai et al., 2016, Lemma 1), we can show that (9) is equivalent to the saddle point problem min V;ˇ2P max”
• Harnessing Infinite-Horizon Off-Policy Evaluation: Double Robustness via Duality (2019, methodology): “Primal-Dual Value Learning Primal-dual optimization techniques have been widely used for off-policy value function learning and policy optimization (Liu et al., 2015; Chen & Wang, 2016; Dai et al., 2017a,b; Feng et al., 2019).”
• Manifold Regularization for Kernelized LSTD (2017, background): “…2) providing theoretical justification, and combining manifold regularization with deep neural nets [25, 26] and other policy evaluation, e.g. , [27, 10] and policy iteration algorithms, 3) learning with a data-dependent kernel that capturing the geometry (equivalent to the manifold regularized…”
• Learning from Conditional Distributions via Dual Embeddings (2016, background): “Please refer the full version [2] for the extension to arbitrary function approximators, e.”
• Local Group Invariant Representations via Orbit Embeddings (2016, background): “The work of [12] considers the general problem of learning from conditional distributions.”