Fast Distribution To Real Regression

By Junier B. Oliva, Willie Neiswanger, Barnab'as P'oczos, Jeff Schneider, Eric P. Xing. 2013

43 citations recorded by the source corpus.

We study the problem of distribution to real-value regression, where one aims to regress a mapping $f$ that takes in a distribution input covariate $P\in \mathcal{I}$ (for a non-parametric family of distributions $\mathcal{I}$) and outputs a real-valued response $Y=f(P) + \epsilon$. This setting was recently studied, and a "Kernel-Kernel" estimator was introduced and shown to have a polynomial rate of convergence. However, evaluating a new prediction with the Kernel-Kernel estimator scales as $\Omega(N)$. This causes the difficult situation where a large amount of data may be necessary for a low estimation risk, but the computation cost of estimation becomes infeasible when the data-set is too large. To this end, we propose the Double-Basis estimator, which looks to alleviate this big data problem in two ways: first, the Double-Basis estimator is shown to have a computation complexity that is independent of the number of of instances $N$ when evaluating new predictions after training; secondly, the Double-Basis estimator is shown to have a fast rate of convergence for a general class of mappings $f\in\mathcal{F}$.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, methodology): “Secondly, compared to density estimation approaches (Póczos et al. 2011, Oliva et al. 2014; 2013), the kernel mean representation is less prone to the curse of dimensionality and admits fast convergence in terms of sample size (Shawe-Taylor and Cristianini 2004, Smola et al. 2007, Grünewälder et…”
Fast Function to Function Regression (2014, background): “This implies, for example, that obtaining estimates for each training instance scales as Ω(nN2), which will be prohibitive for big datasets.”
Estimating Cosmological Parameters from the Dark Matter Distribution (2016, methodology): “If the space of input densities, I, is in a Sobolov ellipsoid type space; see (Ingster & Stepanova, 2011; Laurent, 1996; Oliva et al., 2014) for details.”
Two-stage sampled learning theory on distributions (2014, methodology): “(3)Using a random kitchen sinks approach, with orthonormal basis projection estimators and RBF kernels [19] proposes a distribution regression algorithm that can computationally handle large scale datasets; as with [1], this approach is based on density estimation in R.”
Flexible Transfer Learning under Support and Model Shift (2014, methodology): “We extract raw-pixel features from the images, and use Random Kitchen Sinks [1] to get the coefficients as feature vectors [2], resulting in 2177 features.”
Wasserstein Regression (2020, background): “Other alternatives have been considered for regressing scalar responses on distribution-valued predictors (Póczos et al. 2013; Oliva et al. 2014; Szabó et al. 2016; Bachoc et al. 2017; Thi Thien Trang et al. 2019), but these are either Nadaraya-Watson type kernel estimators that suffer from a severe”