Converting High-Dimensional Regression to High-Dimensional Conditional Density Estimation

By Rafael Izbicki, Ann B. Lee. 2017

93 citations recorded by the source corpus.

There is a growing demand for nonparametric conditional density estimators (CDEs) in fields such as astronomy and economics. In astronomy, for example, one can dramatically improve estimates of the parameters that dictate the evolution of the Universe by working with full conditional densities instead of regression (i.e., conditional mean) estimates. More generally, standard regression falls short in any prediction problem where the distribution of the response is more complex with multi-modality, asymmetry or heteroscedastic noise. Nevertheless, much of the work on high-dimensional inference concerns regression and classification only, whereas research on density estimation has lagged behind. Here we propose FlexCode, a fully nonparametric approach to conditional density estimation that reformulates CDE as a non-parametric orthogonal series problem where the expansion coefficients are estimated by regression. By taking such an approach, one can efficiently estimate conditional densities and not just expectations in high dimensions by drawing upon the success in high-dimensional regression. Depending on the choice of regression procedure, our method can adapt to a variety of challenging high-dimensional settings with different structures in the data (e.g., a large number of irrelevant components and nonlinear manifold structure) as well as different data types (e.g., functional data, mixed data types and sample sets). We study the theoretical and empirical performance of our proposed method, and we compare our approach with traditional conditional density estimators on simulated as well as real-world data, such as photometric galaxy data, Twitter data, and line-of-sight velocities in a galaxy cluster.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/c215beb6ce1740c5a641c1d9a1ce4a0ca7937750

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

(f)RFCDE: Random Forests for Conditional Density Estimation and Functional Data (2019, methodology): “For CDE, we instead choose splits that minimize a loss specific to CDE [Izbicki and Lee, 2017] L(p, p̂) = ∫ ∫ (p(y | x)− p̂(y | x))2 dy dP (x). where P (x) is the marginal distribution of X.”
Semiparametric Counterfactual Density Estimation (2021, methodology): “Alternative approaches could also be used [Díaz and van der Laan, 2011, Hansen, 2004, Izbicki and Lee, 2017], potentially at the expense of some extra computational burden.”
ABC–CDE: Toward Approximate Bayesian Computation With Complex High-Dimensional Data and Limited Simulations (2018, methodology): “…slightly better than NN-KCDE for the same sample, there is an additional improvement in using CDE methods other than kernel smoothers; this difference becomes more pronounced for high-dimensional and complex data (see Izbicki and Lee 2017 for examples, of when traditional kernel smoothers fail).”