Causal Regularization

By Mohammad Taha Bahadori, Krzysztof Chalupka, Edward Choi, Robert Chen, Walter F. Stewart, Jimeng Sun. 2019

57 citations recorded by the source corpus.

I argue that regularizing terms in standard regression methods not only help against overfitting finite data, but sometimes also yield better causal models in the infinite sample regime. I first consider a multi-dimensional variable linearly influencing a target variable with some multi-dimensional unobserved common cause, where the confounding effect can be decreased by keeping the penalizing term in Ridge and Lasso regression even in the population limit. Choosing the size of the penalizing term, is however challenging, because cross validation is pointless. Here it is done by first estimating the strength of confounding via a method proposed earlier, which yielded some reasonable results for simulated and real data. Further, I prove a `causal generalization bound' which states (subject to a particular model of confounding) that the error made by interpreting any non-linear regression as causal model can be bounded from above whenever functions are taken from a not too rich class. In other words, the bound guarantees "generalization" from observational to interventional distributions, which is usually not subject of statistical learning theory (and is only possible due to the underlying symmetries of the confounder model).

Subfields: Optimization & Training, Causal Inference & Discovery

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Causal Inference & Discovery

Citations in the corpus

Toward Causal Representation Learning (2021, background): “10Other dependence measures have been proposed for highdimensional linear settings and time series [28], [119], [121], [122], [126], [226].”
Towards Causal Representation Learning (2021, background): “9Other dependence measures have been proposed for high-dimensional linear settings and time series [124, 226, 27, 122, 119, 125].”
Learning explanations that are hard to vary (2020, background): “Jastrzębski et al. (2017); Zhang et al.”
Causality for Machine Learning (2019, background): “, the irregularities of the two mechanisms add up rather than compensating each other, which would not be the case in the anti-causal direction (for details, see Janzing et al. (2012)).”
CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training (2017, background): “In [3], authors propose a new regularization for training a neural network, which they call causal regularization, in order to assure that the model is predictive in a causal sense.”
A Survey of Learning Causality with Data (2018, background): “Causal regularization [14, 139] is proposed to learn more interpretable and generalizable models.”
TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second (2022, background): “It has been demonstrated that causal knowledge can facilitate various ML tasks, including semisupervised learning, transfer learning and out-of-distribution generalization (Schölkopf et al., 2012; Janzing, 2020; Rothenhäusler et al., 2018).”